All you need to know about Hybrid Auditing in SharePoint 2016

Summary: This blog post will show you how to configure Hybrid Auditing in SharePoint 2016. It will also point out some considerations when deploying this new feature.
Microsoft released a new hybrid feature for SharePoint 2016: Hybrid Auditing. This feature will automatically upload your on-premises user activity logs to Office 365, so administrators can generate reports for users across SharePoint on-premises and Office 365.

* Note: This feature is still in preview! The configuration and capabilities might change in the future.

Configuring Hybrid Auditing

Microsoft did a great job documenting how to configure this new feature for SharePoint 2016. The steps are outlined in this article: https://technet.microsoft.com/en-us/library/mt622371(v=office.16).aspx. I am not going over every step, I am just summarizing the steps below:

  1. Meet the prerequisites
  2. Turn On Audit Log Search Recording
  3. Run the Hybrid picker and select Hybrid Auditing
  4. Check Audit Log Report

Thanks to Vlad Catrinescu who reminded me that you need to restart the Microsoft SharePoint Insights service after patching your SharePoint 2016 farm.

You can do this by running the following PowerShell code:

Add-PSSnapin Microsoft.SharePoint.PowerShell 
Stop-SPService -Identity "Microsoft SharePoint Insights" -IncludeCustomServerRole 
Start-SPService -Identity "Microsoft SharePoint Insights" -IncludeCustomServerRole 

Verify your configuration

If you want to verify that the configuration was successful, here are some tips:

Get Microsoft SharePoint Insights configuration
If you want to make sure that you Hybrid Auditing configuration was done correctly, you can use the following PowerShell cmdlet to see the current configuration:

Get-SPInsightsConfig

This will show you the current configuration for your Hybrid Auditing feature. This might also help you to find any issues you’re facing.

Configure usage and health data collection
In Central Administration, under Monitoring -> Configure usage and health data collection make sure “Enable usage data collection” is checked.
For now I am not sure which checkboxes are required for the hybrid functionality, but these are the ones I have active at the moment.
Hybrid-Auditing-UsageHealth

Troubleshooting

After configuring Hybrid Auditing, I found that I wasn’t getting any on-premises results from the Office 365 Security & Compliance center.
Here you can find some issues that I ran in to and the solution for these issues.

Failed to start a service (Microsoft SharePoint Insights service) that is needed for Hybrid Auditing (Preview) scenario

After running the Hybrid picker I would get this error. It seems that there is a time-out when the picker tries to stop and start the Microsoft SharePoint Insights service. To get around this error, make sure the Microsoft SharePoint Insights service is started from the Services on Server or Services in Farm menu in Central Administration.

I have changed my log location in my on-premises farm
Whenever you change your log location, this change is not automatically picked up by the Microsoft SharePoint Insights service.
Instead, you should run the Hybrid Picker again, then restart the Microsoft SharePoint Insights service from Central Administration.
Your logs will appear in Office 365 after performing these steps.

Results from on-premises aren’t shown in the Office 365 Security & Compliance Center

During my testing, I found that the user mapping isn’t done correctly. As this is a preview feature, I am hoping this will be resolved when the feature will be GA.

I have a user named Kim Akers. In Active Directory, this user is known as:

  • Userprincipalname: kimakers@sharepointrelated.com
  • User logon name (pre-Windows 2000): sprelated\kimakers

Kim works in both SharePoint 2016 (on-premises) and SharePoint Online.
Opening the Security & Compliance Center in the Office 365 Admin Portal, the “Users” field automatically resolves “Kim” as “Kim Akers”. See the screenshot below as a reference:
Hybrid-auditing-kim

The results coming back for this search only show the SharePoint Online activities for Kim.
This is because the results for the on-premises activities for Kim are actually displayed under the user sprelated\kimakers.
hybrid-Auditing-kim-results

Until now, I haven’t found a way to find results for on-premises users directly. The only way to find on-premises activities is to leave the “Users” field empty. This means you will get all results, without any user filter. This makes it hard to find the activities for on-premises users.

If you have any trouble configuring Hybrid Auditing, contact me on Twitter or LinkedIn and I will help wherever I can.

Hybrid features in SharePoint 2013 and 2016

Summary: This post provides an overview of all hybrid SharePoint features that were released by Microsoft for SharePoint 2013 and SharePoint 2016.

During Ignite 2016 in Atlanta, Microsoft released some really cool hybrid features, that I would like to share some information about. The really cool thing about this is that they are not only available for SharePoint 2016, but Microsoft actually made most of them available in SharePoint 2013. The following table will show the availability per feature, so you know which one is available to your environment.

For more information on any specific hybrid feature, click the feature in the table below.

(1) Breaks ALL existing server-to-server trusts. Provider-hosted add-ins are the most commonly found that use server-to-server trust. Make sure to read this blog post for a solution.
(2) There have been major improvements in the CU’s after the initial August 2015 CU for Cloud Hybrid Search. I advise downloading the last CU that has no regressions.

In the last months I have been actively configuring and testing hybrid capabilities in SharePoint 2013. If you have any questions during configuring hybrid features in SharePoint, make sure to contact me on Twitter for the fastest response! I’ll be glad to help with any question.

Fixing apps after configuring SharePoint Hybrid

Update: Microsoft included the fix in the hybrid picker experience. This means you no longer have to perform the steps outlined in this blog post.
You can find the updated article by Microsoft here: https://blogs.technet.microsoft.com/beyondsharepoint/2016/09/15/considerations-when-deploying-sharepoint-office365-hybrid-workloads-in-a-farm-utilizing-provider-hosted-add-ins-or-workflow-manager/

For hybrid search (outbound/inbound query federation or Cloud Hybrid Search Service Application) a manual approach is needed to remediate this scenario.
A KB article was released, which can be found here: https://support.microsoft.com/en-us/help/4010011/provider-hosted-add-ins-stop-working-and-http-401-error


Summary: This article provides a solution to broken provider-hosted add-ins after configuring SharePoint hybrid features. For a full list of hybrid features, see the following article: https://www.sharepointrelated.com/2016/10/04/hybrid-features-sharepoint-2013-and-2016

The following hybrid features will break your server-to-server trusts that were already set up before configuring hybrid for SharePoint 2013 or SharePoint 2016:

This post will describe why this happens and how we can fix this.

In order to establish a server-to-server trust between your on-premises SharePoint environment and Office 365, Microsoft relies on the SPAuthenticationRealm. More information can be found here: https://technet.microsoft.com/en-us/library/jj219756.aspx.

This article has a “Caution” section, warning that any access tokens created for a specific realm, won’t work after changing the SPAuthenticationRealm.
SharePoint hybrid

To fix this, I wrote a script that gives you 2 options:

Undo Fix
Reverse the changes made by configuring Hybrid. It will change the SPAuthenticationRealm back to the old value. All SharePoint hybrid features stop working. All your provider-hosted add-ins will work again. This option will try to change your SPTrustedSecurityTokenIssuers so that it uses the new SPAuthenticationRealm set by configuring hybrid.

CautionThere are some notes that I described later in this post, make sure to read them.

Running the script will result in something like this:
SharePoint hybrid
Running the Fix-Hybrid.ps1 script

You can download the script here:
Fix-Hybrid

Notes
If you choose to fix your SPTrustedSecurityTokenIssuers, you will need to do some additional work to have everything work again.

  • Regrant app permissions

App permissions rely on the SPAuthenticationRealm.
This means that any App permissions that you set, will be gone after updating your SPTrustedSecurityTokenIssuers.
You will have to register the apps again and assign the permissions to the app.
The following script can do this for you (the current script is app-instance based, this means you have to run it for every app instance.
Also, make sure to change the variables in the script before running it.
Set-SPAppPermissions

  • Workflow Manager

Workflow Manager also relies on the SPAuthenticationRealm. Thanks to Ruben de Boer for proposing the solution.
After running the Fix-Onboarding.ps1 script, make sure to remove the existing Workflow Service Application Proxy.
Then run the Register-SPWorkflowService cmdlet again. Make sure to use the same scope that you used before. I recommend using the -Force parameter.

I hope this helps anyone! Do not hesitate to contact me if you have any trouble using the script of have any questions.

Cloud hybrid search considerations

Summary: This blog post describes some limitations that you need to consider before implementing cloud hybrid search in your organization.

Do not hesitate to contact me on twitter or LinkedIn to see if there are any changes that are not reflected in this blog post.
For a full overview of how Cloud Hybrid Search works, read my blog post: Everything you need to know about Cloud Hybrid Search

1. Provider-hosted apps
All provider-hosted add-ins will break when running the onboarding script *
One part of the onboarding script will change the SPAuthenticationRealm for your entire SharePoint farm.
As all your SPTrustedSecurityTokenIssuers rely on this SPAuthenticationRealm, they stop working after running the onboarding script.

Microsoft released a new hybrid picker that includes a fix for this issue. If you are configuring Cloud Hybrid Search only using the onboarding script, you have to do some additional work to get your apps to work again. Microsoft released a KB article with the fix here: https://support.microsoft.com/en-us/help/4010011/provider-hosted-add-ins-stop-working-and-http-401-error

2. Search customizations
The Cloud Search Service Application shares a similar architecture as the native SharePoint Search Service Application.
However, customization is limited because the search experience is derived from Office 365.

Below is a table that shows the current limitations in search customizations when using Cloud Hybrid Search.
Hybrid Cloud Search customizations

3. Searching on Non-default Zone URL’s
From: https://blogs.msdn.microsoft.com/spses/2016/07/19/sharepoint-2016-hybrid-search-stuff-you-should-know-about-cloud-ssa/

Crawling Happens on-premises and the Default Zone url which you are crawling is fed to the Index on the Cloud. Since the Query is being served from Components in the Cloud which have no knowledge of the alternate URL’s available for On-prem Web-application, no URL Translation happens.
In Fact there is No concept of AAM’s in cloud Space.

The Only way to work around these limitations is to ensure that:

  1. The Public Default Zone URL should be a FQDN URL or The END URL which you want to Show up to the users in search results.
  2. The Default Zone URL which is being crawled should have Windows Claims Auth so that ACL information sent along with crawled Content is recognized by Search Components correctly.
  3. The User Needs to Perform Search from a URL while logged in windows Identity , so that it can be resolved effectively to validate Claims & Security Trimming to work effectively.

Note : Please ensure that Public URL which you want end users to see , should be accessible to Crawler on premise to be able to crawl the site successfully.

4. Windows Claims Only
From: https://blogs.msdn.microsoft.com/spses/2016/07/19/sharepoint-2016-hybrid-search-stuff-you-should-know-about-cloud-ssa/

Cloud SSA or To be Specific the Search Components in SharePoint online at present do not understand any other ACL type other than Windows Claims , This ACL information ,about the Content which is fed along the index is used for Security Trimming the results . SAML Identity Providers are not supported with Cloud SSA . So basically the On-prem Site you are crawling using the Cloud SSA should be using Windows Auth only .

if you search on an extended URL of Web-applications which is ADFS / LDAP or Some other type of Auth other than Windows Claims , No search results will be returned as the logged in user’s Identity cannot be resolved to ACL mapping we have on an Index Item.

5. Index item limits and pricing
Vlad Catrinescu blogged about this on his blog.
For every 1TB of pooled storage in SharePoint Online, we are allowed to put one million index items from our On-Premises SharePoint Farm.

You can check your current searchable items from the Search Service Application in Central Administration
Cloud hybrid search considerations

In this example, we would need 13TB of pooled storage in SharePoint Online. This might mean that you have to reconfigure your content sources.

6. Security and compliancy
As your index is stored in Office 365, what does this mean compliancy wise?
This is the answer from Microsoft:

The content that is passed from on-premises to the azure cloud search connector (SCS) consists of crawled properties, keywords, ACLs, tenant info and some other metadata about the item. This is encrypted on-premises using a key supplied by the SCS and transmitted to the endpoint in Azure. Once there it is stored in an encrypted blob store and queued for processing. We retain the encrypted package in the blob store for use should we need to issue a content recrawl. The encrypted object is not the document though, it is just a parsed and filtered version that makes sense to the search engine.

In my opinion it would be wise to consult with your legal department before setting up Cloud Hybrid search to make sure it is okay to store content in the cloud.
You can always modify the content sources to exclude highly classified documents.

7. Licensing and Office 365 accounts
Every user that wants to use the new Cloud Hybrid Search functionality will need an active Office 365 license and an account synchronized from your on-premises Active Directory.
Cloud-Hybrid-Search-Identity
As items are indexed in Office 365, the access control entities are looked up in the cloud directory service.
Hybrid-Search-FederatedAccount
User SIDs are mapped to PUIDs; Group SIDs are mapped to Object IDs; and and are mapped to .

8. Documentation
As the cloud hybrid search is still in preview, documentation is limited at best.
For example, if you are using a proxy for Internet access on your server, make sure to specify this in your machine.config.
I have created a more detailed blog post around this issue here: https://www.sharepointrelated.com/2015/12/11/cloud-hybrid-search-proxy-settings/

The documentation for Cloud Hybrid Search has been greatly improved. You can find all information you need here: https://technet.microsoft.com/en-us/library/dn720906.aspx

Office 365 Video: My first impression

I’ve had the opportunity to look into the possibilities for organizations to use Office 365 Video. Using videos within an organization can be a great way to connect to your employees. This could be a welcome video from the new CEO, a safety regulation video or training material to help new employees get on their way.

Homepage
The Video portal homepage consists of a few key sections. The Spotlight area gives you the possibility to show videos, which have a strong presence on the homepage. The Popular videos section shows videos that are popular within your organization. The last section is the Spotlight channels. Here you can select 3 channels, from which you show the spotlight videos on your homepage.

Office 365 Video portal settings
Office 365 Video portal settings

The popular videos in particular is a great way to see which videos are popular in your organization. The only downside is that an administrator cannot remove or influence the popular video section.
For each video on the homepage you can see the total views and the length of the video.

Personally, I would like to have a little more control over the Office 365 Video homepage. For instance, a channel overview on the left or right side of the screen would be useful to navigate between channels. The possibility to view a video from the homepage could be easier as well, as now you have to click the context action menu (the 3 dots). A pop-up will be shown, starting the video.

Channels
Besides showing videos on your homepage, an administrator has the possibility to create channels. Permissions can be set per channel to allow only certain people (or groups) to view the videos in the channel.

When an administrator creates a channel, a SharePoint Online site collection is created in the background. This is where the video and all metadata will be saved. A copy will be sent to Azure Media services. Microsoft posted this nice video on YouTube explaining how this works behind the scene.

Start a new channel
Start a new channel

The channel creation process could be more smoothly, as the administrator is required to wait for the site collection to be created, which takes some time. When you close the window while a channel is creating, the channel will still be created, but the channel color you selected won’t be saved. It would be nice if the creation could happen behind the screens, so the administrator doesn’t have to wait for the site collection to be provisioned.

Video
While watching a video, Office 365 Video provides you with several options. You can share the video, see the description of the video, or even comment on the video by using the inline Yammer conversation on the right side of the screen. Below the video is a list of videos that you may also like.

Video Yammer conversations
Inline Yammer conversations

Videos will be processed and displayed in Delve as well. This is a really neat feature that helps employees engage with Office 365 Video without actually navigating to Office 365 Video in the first place, as relevant videos are being showed in Delve.

Conclusion
In conclusion: Office 365 Video is a great initiative that has a lot of potential. At the moment of this writing, I think the service could still be improved by integrating Office 365 Video with other Office 365 services even more. In particular, some of the features that SharePoint online has, could help make Office 365 Video even better. As Office 365 Video is still a relatively new service, I expect Microsoft will make regular improvements that will add business value organizations.

If you are currently testing or using Office 365 Video, make sure to provide feedback to help improve the service. On the top right corner of each page you will find the option to send Feedback to Microsoft.

feedback
Feedback