Category Archives: Exchange

Deploying MailItemsAccessed Audit Event in Office 365

MailItemsAccessed is a new audit event in Office 365 that records when email data is accessed by mail protocols and clients.

Why is MailItemsAccessed so important?

During an investigation where a mailbox has been accessed by an unauthorized party, there are often legal requirements (State, Federal and Global Treaties such as GDPR) to notify individuals if their personally identifiable information was accessed. Without MailItemsAccessed we could only say that the attacker had the capability of accessing all mailbox contents, but we couldn’t say which exact emails were accessed. The sync event is still not as definitive as we would like, but it does show that the attacker now has possession of the mailbox contents. If the attacker accessed the mailbox via a web browser, then it’s helpful to know which individual items were accessed.

If a privileged account was compromised, it’s also a good idea to check whether the attacker enabled the Bypass audit log PowerShell command to cover their tracks.

For more details, see Access to crucial events for investigations.

How does MailItemsAccessed compare to MessageBind?

MailItemsAccessed replaces the audit event ‘MessageBind’ which was deprecated in Exchange Online on 1/23/2019. This audit event began rolling out in Q1 2020 after a 12 month pause from the first announcement in January 2019. Tony Redmond has documented the history of this rollout on his blog, with his latest post on March 6, 2020 (here).

MessageBind was only available for the AuditAdmin user logon type and did not record actions performed by the Mailbox Owner or Delegate. MessageBind only covered actions by a mail client and did not record sync activities. MessageBind also generated multiple audit records for the same email message. MailItemsAccessed fixes all these deficiencies.

MailItemsAccessed applies to all logon types (Owner, Admin and Delegate)

MailItemsAccessed applies to both an individual email being read in addition to a ‘sync’ event such as MAPI, POP or IMAP downloading all email in a client.

MailItemsAccessed aggregates multiple events into fewer audit records.

Licensing

Office E5 or (M365 E3 + “E5 Compliance” add-on)

One question that is often asked is: “If I buy just one license, does this enable the capability for all users.” The answer is no. Only users with the appropriate license will have the MailItemsAccessed logged.

Deployment

MailItemsAccessed is only enabled by default when the E5 feature “Microsoft 365 Advanced Auditing” license has been applied to the account.

clip_image002

(In PowerShell the auditing license will appear as “M365_ADVANCED_AUDITING”).

To find out how many of your current mailboxes are logging the MailItemsAccessed event run this Exchange Online PowerShell command:

get-mailbox -ResultSize unlimited | where {$_.AuditOwner -like ‘*Accessed*’}

Note: See troubleshooting section if you have the right license and MailItemsAccessed is still not appearing.

Prior to 2/1/2019, Mailbox Owner auditing only logged a single event by default: MailboxLogin. After 2/1/2019, additional events were added unless auditing had been customized. If you customized the mailbox actions to audit for any logon type before mailbox auditing on by default was enabled in your organization, the customized settings are preserved on the mailbox and aren’t overwritten by the default mailbox actions that were since added. The exception to this rule seems to be with MailItemsAccessed because it was appended to the E5 mailboxes that had been set to use customized audit events.

To reset auditing to defaults you can run use the DefaultAuditSet parameter, which is generally recommended because according to the documentation “Any new mailbox actions that are released by Microsoft are automatically added to the list of audited actions for the logon type.”

Set-Mailbox -Identity (mailbox identity) –DefaultAuditSet Admin,Delegate,Owner

If you want to find out the mailboxes that have the default auditing enabled, run this command and look for “Admin, Delegate, Owner.” The absence of one of these means that audit customizations was applied to that mailbox.

get-mailbox | select name,DefaultAuditSet

clip_image004

For example, in the screen shot above (1) means that all three roles were customized and (2) means only the Owner was customized because it is absent from the list.

Note: DefaultAuditSet does not audit all possible audit events. It will audit the following seven events (or eight when an E5 license is applied to the mailbox)

1. Update

2. MoveToDeletedItems

3. SoftDelete

4. HardDelete

5. UpdateFolderPermissions

6. UpdateInboxRules

7. UpdateCalendarDelegation

8. MailItemsAccessed (<- THIS IS ONLY ADDED IF AN E5 License is applied to the mailbox)

For a list of the defaults see the documentation (here).

The following three events can be added as additional audit events for the Owner logon type:

1. Create

2. MailboxLogin

3. Move

Therefore, to apply all 11 possible audit events, run this command:

get-mailbox -ResultSize unlimited | set-mailbox -AuditOwner @{Add= “Update”,”MoveToDeletedItems”,”SoftDelete”,”HardDelete”,”UpdateFolderPermissions”,”UpdateInboxRules”,”UpdateCalendarDelegation”,”Create”,”MailboxLogin”,”Move”,”MailItemsAccessed”}

Repeat above command and swap out AuditOwner for AuditDelegate and AuditAdmin but remember to check the table (here) because not all audit events are available for Admin

Note: A user with full mailbox access to another user’s mailbox is logged as AuditDelegate.

Searching Audit Events

You can search for MailItemsAccessed events in Protection.office.com

image

Compliance.microsoft.com will eventually replace Protection.office.com.

image

You can also use Exchange Online PowerShell to search for audit events, which is required if you need to search for events older than 90 days. Search-UnifiedAuditLog -Operations MailItemsAccessed or Search-MailboxAuditLog -Operations MailItemsAccessed

Other useful Exchange Online PowerShell commands:

View which events are being logged on a single mailbox (“Joe”)

get-mailbox joe | select -ExpandProperty auditadmin

get-mailbox joe | select -ExpandProperty auditowner

get-mailbox joe | select -ExpandProperty auditdelegate

Report how many accounts have auditing enabled:

get-mailbox | group-object AuditEnabled

Enable auditing on all mailboxes and increase audit log retention from 90 days to 180 days:

get-mailbox -resultsize unlimited | set-mailbox -AuditEnabled $true -AuditLogAgeLimit 180

Audit Log retention is independent of whether or not a retention policy (aka Legal hold) is applied to the mailbox. For example, if a mailbox is under legal hold, the audit events are not retained longer than the duration set by the AuditLogAgeLimit parameter.

If you increase the age beyond 90 days, you can only find those items in PowerShell using Search-MailboxAuditLog.

The following capacity limitations apply to mailbox auditing:

– No more than 3 million audit records are allowed per mailbox

– No more than 30 GB of audit records are allowed per mailbox (100GB if legal hold or retention policy has been applied to the mailbox)

– Tony also states in his blog “Exchange Online applies throttling for MailItemsAccessed events. If a mailbox generates more than 1,000 bind events in a 24-hour period, Exchange Online stops recording MailItemsAccessed events for bind operations for another 24 hours before resuming capture of these events. Microsoft says that less than 1% of mailboxes are subject to throttling.”

Troubleshooting

Assume you have a mailbox where MailItemsAccessed is not applied, but the mailbox has an E5 license.

clip_image010

You then try to add the audit event but you get an error that its only available for users with the appropriate license.

clip_image012

Double-check to see that you have the “Microsoft 365 Advanced Auditing” license type assigned.

clip_image013

Note: In my case, even though the box was checked, it did not work because this license assignment was inherited from an Azure AD P1 feature called Group-based licensing. So to work-around this bug, I directly assigned the license via PowerShell (since I couldn’t via the GUI since the checkbox was already selected) and that allowed the MailItemsAccessed to be applied.

clip_image015

$MSOLSKU = “(tenantname):ENTERPRISEPREMIUM”

$myO365Sku1 = New-MsolLicenseOptions -AccountSkuId $MSOLSKU

Set-MsolUserLicense -UserPrincipalName (username) -LicenseOptions $myO365Sku1

Top 10 Fixes for troubleshooting free/busy between Exchange on-premises and Exchange Online in Office 365

Free/busy often fails to work out-of-the-box after configuring Hybrid Exchange with Office 365. Here are my top ten fixes:

 

  1. Set the sharing policy to match on-premises and cloud.

    First, Connect to Exchange Online Remote Powershell and run get-sharingpolicy

    Then connect to on-premises Exchange Management Shell and run get-sharing policy

    Then make the two match on both sides.

 

Set-SharingPolicy -Identity SharingPolicy01 -Domains ‘contoso.com: CalendarSharingFreeBusySimple’, ‘atlanta.contoso.com: CalendarSharingFreeBusyReviewer’, ‘beijing.contoso.com: CalendarSharingFreeBusyReviewer’

 

  1. Set the organization relationship domains to include all accepted domains on both on-premises and cloud (always requires an IISRESET for it to take effect)
    This script helps identify missing domains in an existing relationship:

     

    if ( (Get-OrganizationRelationship).DomainNames -contains (Get-Mailbox user).PrimarySmtpAddress.Domain) { write-host “The domain was found” -ForegroundColor Green } else { write-host (Get-Mailbox user).PrimarySmtpAddress.Domain “was not found” -ForegroundColor Yellow}

     

    $OrgRel = Get-OrganizationRelationship Contoso

    $OrgRel.DomainNames += “contoso.com”

    Set-OrganizationRelationship $OrgRel.Name -DomainName $OrgRel.DomainNames

     

     

    1. If the autodiscover DNS name is not published in external DNS, and if the client doesn’t want to do that, then manually configure TargetSharingEpr to use the published EWS path

      Get-OrganizationRelationship -Identity “O365 to On-premises – (GUID)” | Set-OrganizationRelationship -TargetSharingEpr https://mail.contoso.com/ews/exchange.asmx

    4) For ‘401 errors’ try disabling the IOC connector in Exchange 2013 to have oAuth fall back to dAuth


    5) Sometimes it’s necessary to set the on-premises EWS virtual directory “WSSecurityAuthentication” value back to defaults (some clients change this if they do load balanced CAS)
    (this is commonly a last resort)

    Need to change WSSecurityAuthentication to False for EWS Virtual directory.

        a.       Set-WebServicesVirtualDirectory “Exch CAS\ews*” –WSSecurityAuthentication $false

        b.      Need to Stop MSExchangeServicesAppPool.

        c.       Need to Start  MSExchangeServicesAppPool.

     

      Need to change WSSecurityAuthentication to True again for EWS Virtual Directory.

        a.       Set-WebServicesVirtualDirectory “Exch CAS\ews*” –WSSecurityAuthentication $True

        b.      Need to Stop MSExchangeServicesAppPool.

        c.       Need to Start  MSExchangeServicesAppPool.

     

      Need to change WSSecurityAuthentication to False for Autodiscover Virtual directory.

        a.       Set-AutodiscoverVirtualDirectory “Exch CAS\Auto*” –WSSecurityAuthentication $false

        b.      Stop MSExchangeAutodiscoverAppPool.

        c.       Start  MSExchangeAutodiscoverAppPool.

     

      Change WSSecurityAuthentication to True again for Autodiscover Virtual Directory.

        a.       Set-AutodiscoverVirtualDirectory “Exch CAS\Auto*” –WSSecurityAuthentication $true

        b.      Stop MSExchangeAutodiscoverAppPool.

        c.       Start  MSExchangeAutodiscoverAppPool.

     

    6) If the Exchange Server is behind a web proxy then it is usually necessary to configure InternetWebProxy Set-ExchangeServer <Server Name> -InternetWebProxy:http://<Proxy Address>:<Proxy Port>

     

    7)  Verify the availability address space and see required SMTP domain with access method.

        Get-AvailabilityAddressSpace (Run this on-prem)

     

    8) Try running diagnostic commands:
    You can also use the Test-FederationTrust (on prem only) and Test-OrganizationRelationship  (run this both on prem and in cloud too)

    And you can also use this website to run tests: https://www.testexchangeconnectivity.com/

    9) Make sure that the cloud user you are searching for has a valid (tenant).mail.onmicrosoft.com alias on their target mailbox (make sure Azure AD Connect is properly replicating that attribute, and/or, that the Exchange Address Policy is not blocking inheritance on that particular user/object).

     

    10) Run these commands to gather diagnostic information:

    Onpremises:

    Start-Transcript

    Get-FederationTrust | fl

    Get-FederatedOrganizationIdentifier | fl

    Get-OrganizationRelationship | fl

    Get-WebServicesVirtualDirectory | Export-Clixml C:\temp\WebVdir.xml

    Get-AutoDiscoverVirtualDirectory | Export-Clixml C:\temp\AutoDVdir.xml

    Get-RemoteMailbox bobc_sync | fl

    Get-Mailbox “on-premises John Doe User” | fl

    Test-FederationTrust -UserIdentity [email protected] | fl

    Test-FederationTrustCertificate | fl

    Get-IntraOrganizationConnector | fl

    Stop-Transcript

     

    Online:

    Start-Transcript

    Get-FederationTrust | fl

    Get-FederatedOrganizationIdentifier | fl

    Get-OrganizationRelationship | fl

    Get-MailUser “on-premises John Doe User” | fl

    Get-Mailbox “Cloud user” | fl

    Get-IntraOrganizationConnector | fl

    get-OrganizationRelationship | Test-OrganizationRelationship -UserIdentity “cloud user”

    Stop-Transcript

     

     

     

    And when all else fails I reference these two blog articles:

    https://blogs.technet.microsoft.com/exchange/2018/02/06/demystifying-hybrid-freebusy-what-are-the-moving-parts/

    and

    https://blogs.technet.microsoft.com/exchange/2018/03/02/demystifying-hybrid-freebusy-finding-errors-and-troubleshooting/

How to fix Exchange Online Hybrid Outbound Connector 454 4.7.5 Certificate Validation Failure

While recently helping a client setup an Exchange Hybrid, the cloud to on-premises mail flow was failing validation due to 454 4.7.5 Certificate Validation Failure.

The next step was to verify that the TlsCertificateName value was properly set on the send and receive connectors to match the certificate name, following these articles:

https://blogs.technet.microsoft.com/lalitbisht/2017/06/03/mailflow-issue-from-exchange-on-prem-to-office-365/

https://practical365.com/exchange-server/configuring-the-tls-certificate-name-for-exchange-server-receive-connectors/

In this case, the TlsCertificateName was already set correctly to match the certificate name (the Hybrid Exchange Wizard does a good job at setting that correctly).

The next step was to enable Verbose logging on the on-premises receive connector so that we can get a better look at the error.

To save time, I restarted the “Microsoft Exchange Frontend Transport” service so that the logging would take effect sooner.

Then navigating to the log directory can be a bit tricky:

C:\Program Files\Microsoft\Exchange Server\v15\TransportRoles\Logs\FrontEnd\ProtocolLog\SmtpReceive

Opening up the file revealed a very helpful bit of information! The SSL Certificate that Microsoft Office 365 is presenting to the Exchange server for the TLS encrypted email is not a trusted root. How can this be?

 

To cut to the chase, the root cause was that the server had not had windows updates run in a LONG time and therefore was really far behind in its root certificates.

The least disruptive solution was to download the Office certificate chains from Microsoft (here) and install them on the on-premises Exchange Server. Then after restarting the “Microsoft Exchange Frontend Transport” service and waiting a few minutes, the validation was successful.

PowerShell script to automatically heal non-deliverable emails (NDRs) as X500

One of the possible causes for a non-delivery report (NDR) is when a mail object in Exchange is moved or removed (Contact, Mailbox, MailUser, etc). When an object is moved, any internal user who had previously emailed that object will have a cached entry in their autocomplete cache that no longer matches up to what now exists. This happens because the autocomplete cache stores the value of the LegacyExchangeDN attribute of the original object before it is moved. When an object is moved, a new LegacyExchangeDN is created.

When sending a message, Outlook will check the Global Address List (GAL), and if it can’t find a match in the GAL, IMCEA encapsulation is used (Internet Mail Connector Encapsulated Addressing). An IMCEA encapsulated address looks like:

IMCEAEX-_O=[email protected]contoso.com

So when an email is sent to the original cached object, an NDR will be sent back to the user containing a construct of the LegacyExchangeDN that it failed to reach, in a slightly different format:

A trailing “EX” at the end of IMCEAEX indicates that a non-SMTP address was encapsulated.

While a quick fix is to have the sender clear their autocomplete cache and re-send the message, a more automated solution is desirable when there are hundreds of potential senders who cached the old object.

I discovered a nifty PowerShell script (here) written by Michael England on 2/11/2013 that searches the Message Tracking Logs for NDRs, and then reconstructs the LegacyExchangeDN from the IMCEA format and adds that as an X500 proxy alias on the target recipient object (if it can be found). This works great in a scenario where the contact object was replaced by a Mailbox or MailUser object.

One of the things I appreciate about how he wrote this script is that when you run the script it reports what would be modified, then when you are ready to modify you just add the -Autoheal parameter to the end of the script. This way you can get an idea of what would be modified before it happens.

I had to modify Michael’s script because Microsoft changed the behavior of the LegacyExchangeDN value in Exchange 2010 SP1 Rollup 6 (released 10/27/2011) to add 3 random hex characters for uniqueness at the end of the LegacyExchangeDN. So when I attempted to use Michael’s script, it was not finding the destination object to add the X500 to it because the 3 random characters threw the search off. So I have posted a very minor change to an otherwise awesome script to strip the 3 characters when searching for the object to place the proxy alias on.

            Write-Host “looking for: ” $user.Substring(0,$user.Length-3)

            $user = $user.Substring(0,$user.Length-3)

 

I also made three separate copies, depending on the recipient object type (Mailbox, MailContact or MailUser).

You can download my slightly modified version of Michael’s script from the Microsoft Technet Gallery here.

https://gallery.technet.microsoft.com/Search-Message-Tracking-6be6d1b7

I left all original credit to Michael in the script, since I only slightly changed the code to strip the 3 random hex characters out, and made separate copies of the script to search for MailUser and MailContact since his script only searched for Mailboxes. I was stoked to discover this awesome script by Michael.

Also shout out to a different Michael, Michael de Rooij for an excellent blog article on this topic here: https://eightwone.com/2013/08/12/legacyexchangedn-attribute-myth/