All you need to know about Hybrid Auditing in SharePoint 2016

Summary: This blog post will show you how to configure Hybrid Auditing in SharePoint 2016. It will also point out some considerations when deploying this new feature.
Microsoft released a new hybrid feature for SharePoint 2016: Hybrid Auditing. This feature will automatically upload your on-premises user activity logs to Office 365, so administrators can generate reports for users across SharePoint on-premises and Office 365.

* Note: This feature is still in preview! The configuration and capabilities might change in the future.

Configuring Hybrid Auditing

Microsoft did a great job documenting how to configure this new feature for SharePoint 2016. The steps are outlined in this article: https://technet.microsoft.com/en-us/library/mt622371(v=office.16).aspx. I am not going over every step, I am just summarizing the steps below:

  1. Meet the prerequisites
  2. Turn On Audit Log Search Recording
  3. Run the Hybrid picker and select Hybrid Auditing
  4. Check Audit Log Report

Thanks to Vlad Catrinescu who reminded me that you need to restart the Microsoft SharePoint Insights service after patching your SharePoint 2016 farm.

You can do this by running the following PowerShell code:

Add-PSSnapin Microsoft.SharePoint.PowerShell 
Stop-SPService -Identity "Microsoft SharePoint Insights" -IncludeCustomServerRole 
Start-SPService -Identity "Microsoft SharePoint Insights" -IncludeCustomServerRole 

Verify your configuration

If you want to verify that the configuration was successful, here are some tips:

Get Microsoft SharePoint Insights configuration
If you want to make sure that you Hybrid Auditing configuration was done correctly, you can use the following PowerShell cmdlet to see the current configuration:

Get-SPInsightsConfig

This will show you the current configuration for your Hybrid Auditing feature. This might also help you to find any issues you’re facing.

Configure usage and health data collection
In Central Administration, under Monitoring -> Configure usage and health data collection make sure “Enable usage data collection” is checked.
For now I am not sure which checkboxes are required for the hybrid functionality, but these are the ones I have active at the moment.
Hybrid-Auditing-UsageHealth

Troubleshooting

After configuring Hybrid Auditing, I found that I wasn’t getting any on-premises results from the Office 365 Security & Compliance center.
Here you can find some issues that I ran in to and the solution for these issues.

Failed to start a service (Microsoft SharePoint Insights service) that is needed for Hybrid Auditing (Preview) scenario

After running the Hybrid picker I would get this error. It seems that there is a time-out when the picker tries to stop and start the Microsoft SharePoint Insights service. To get around this error, make sure the Microsoft SharePoint Insights service is started from the Services on Server or Services in Farm menu in Central Administration.

I have changed my log location in my on-premises farm
Whenever you change your log location, this change is not automatically picked up by the Microsoft SharePoint Insights service.
Instead, you should run the Hybrid Picker again, then restart the Microsoft SharePoint Insights service from Central Administration.
Your logs will appear in Office 365 after performing these steps.

Results from on-premises aren’t shown in the Office 365 Security & Compliance Center

During my testing, I found that the user mapping isn’t done correctly. As this is a preview feature, I am hoping this will be resolved when the feature will be GA.

I have a user named Kim Akers. In Active Directory, this user is known as:

  • Userprincipalname: kimakers@sharepointrelated.com
  • User logon name (pre-Windows 2000): sprelated\kimakers

Kim works in both SharePoint 2016 (on-premises) and SharePoint Online.
Opening the Security & Compliance Center in the Office 365 Admin Portal, the “Users” field automatically resolves “Kim” as “Kim Akers”. See the screenshot below as a reference:
Hybrid-auditing-kim

The results coming back for this search only show the SharePoint Online activities for Kim.
This is because the results for the on-premises activities for Kim are actually displayed under the user sprelated\kimakers.
hybrid-Auditing-kim-results

Until now, I haven’t found a way to find results for on-premises users directly. The only way to find on-premises activities is to leave the “Users” field empty. This means you will get all results, without any user filter. This makes it hard to find the activities for on-premises users.

If you have any trouble configuring Hybrid Auditing, contact me on Twitter or LinkedIn and I will help wherever I can.

Hybrid features in SharePoint 2013 and 2016

Summary: This post provides an overview of all hybrid SharePoint features that were released by Microsoft for SharePoint 2013 and SharePoint 2016.

During Ignite 2016 in Atlanta, Microsoft released some really cool hybrid features, that I would like to share some information about. The really cool thing about this is that they are not only available for SharePoint 2016, but Microsoft actually made most of them available in SharePoint 2013. The following table will show the availability per feature, so you know which one is available to your environment.

For more information on any specific hybrid feature, click the feature in the table below.

(1) Breaks ALL existing server-to-server trusts. Provider-hosted add-ins are the most commonly found that use server-to-server trust. Make sure to read this blog post for a solution.
(2) There have been major improvements in the CU’s after the initial August 2015 CU for Cloud Hybrid Search. I advise downloading the last CU that has no regressions.

In the last months I have been actively configuring and testing hybrid capabilities in SharePoint 2013. If you have any questions during configuring hybrid features in SharePoint, make sure to contact me on Twitter for the fastest response! I’ll be glad to help with any question.

Fixing apps after configuring SharePoint Hybrid

Update: Microsoft included the fix in the hybrid picker experience. This means you no longer have to perform the steps outlined in this blog post.
You can find the updated article by Microsoft here: https://blogs.technet.microsoft.com/beyondsharepoint/2016/09/15/considerations-when-deploying-sharepoint-office365-hybrid-workloads-in-a-farm-utilizing-provider-hosted-add-ins-or-workflow-manager/

For hybrid search (outbound/inbound query federation or Cloud Hybrid Search Service Application) a manual approach is needed to remediate this scenario.
A KB article was released, which can be found here: https://support.microsoft.com/en-us/help/4010011/provider-hosted-add-ins-stop-working-and-http-401-error


Summary: This article provides a solution to broken provider-hosted add-ins after configuring SharePoint hybrid features. For a full list of hybrid features, see the following article: https://www.sharepointrelated.com/2016/10/04/hybrid-features-sharepoint-2013-and-2016

The following hybrid features will break your server-to-server trusts that were already set up before configuring hybrid for SharePoint 2013 or SharePoint 2016:

This post will describe why this happens and how we can fix this.

In order to establish a server-to-server trust between your on-premises SharePoint environment and Office 365, Microsoft relies on the SPAuthenticationRealm. More information can be found here: https://technet.microsoft.com/en-us/library/jj219756.aspx.

This article has a “Caution” section, warning that any access tokens created for a specific realm, won’t work after changing the SPAuthenticationRealm.
SharePoint hybrid

To fix this, I wrote a script that gives you 2 options:

Undo Fix
Reverse the changes made by configuring Hybrid. It will change the SPAuthenticationRealm back to the old value. All SharePoint hybrid features stop working. All your provider-hosted add-ins will work again. This option will try to change your SPTrustedSecurityTokenIssuers so that it uses the new SPAuthenticationRealm set by configuring hybrid.

CautionThere are some notes that I described later in this post, make sure to read them.

Running the script will result in something like this:
SharePoint hybrid
Running the Fix-Hybrid.ps1 script

You can download the script here:
Fix-Hybrid

Notes
If you choose to fix your SPTrustedSecurityTokenIssuers, you will need to do some additional work to have everything work again.

  • Regrant app permissions

App permissions rely on the SPAuthenticationRealm.
This means that any App permissions that you set, will be gone after updating your SPTrustedSecurityTokenIssuers.
You will have to register the apps again and assign the permissions to the app.
The following script can do this for you (the current script is app-instance based, this means you have to run it for every app instance.
Also, make sure to change the variables in the script before running it.
Set-SPAppPermissions

  • Workflow Manager

Workflow Manager also relies on the SPAuthenticationRealm. Thanks to Ruben de Boer for proposing the solution.
After running the Fix-Onboarding.ps1 script, make sure to remove the existing Workflow Service Application Proxy.
Then run the Register-SPWorkflowService cmdlet again. Make sure to use the same scope that you used before. I recommend using the -Force parameter.

I hope this helps anyone! Do not hesitate to contact me if you have any trouble using the script of have any questions.

Restore deleted site collections SharePoint 2013

In SharePoint 2013 it is possible to restore a deleted site collection. For more information, read this article: http://technet.microsoft.com/en-us/library/hh272537.aspx

You can use the Restore-SPDeletedSite cmdlet to restore a site collection.

However, if you removed the site collection using the Remove-SPSite cmdlet using PowerShell, the site collection will not be stored in a SPDeletedSite object.

This means you cannot restore a site collection that has been removed using PowerShell.

 

Add PDF mimetype for all Web Applications oneliner

By default, PDF files cannot be opened directly from SharePoint 2010/SharePoint 2013.

To add the PDF mimetype to all Web Applications (Instead of doing it seperately for each Web Application), you can use the following oneliner:

Get-SPWebApplication | %{$_.AllowedInlineDownloadedMimeTypes.Add("application/pdf");$_.Update()}

Current cache accounts PowerShell oneliner

This oneliner will show you the currently configured cache accounts for each of your Web Applications. Just paste this line of code in your SharePoint Management Shell:

Get-SPWebApplication | %{Write-Host "Web Application: " $_.url "`nSuper user: " $_.properties["portalsuperuseraccount"] "`nSuper reader: " $_.properties["portalsuperreaderaccount"] "`n"}

This is the output for the above command in my test environment:

Current cache accounts PowerShell oneliner

In my case, I have not properly configured the super user accounts for all my Web Applications.

Be aware that misconfigured super user accounts can lead to access problems. You may not be able to access your SharePoint Web Application!

Get Site Collections per Content Database oneliner

This will be a very short post, as it contains only 1 line of PowerShell code, and is not very hard to understand. It shows you all content databases, and for each of the content databases, it will show you the site collections within this content database.

Get-SPContentDatabase | %{Write-Output "- $($_.Name)”; foreach($site in $_.sites){write-Output $site.url}}

That wasn’t so bad right? Just copy and paste this code into your SharePoint Management Shell (As administrator), and let PowerShell do your work for you!

Get Site Collections per Content Database oneliner

Now let’s say you want to save this output to a .txt file, you can add

> C:sitecollections.txt

The full command would be:

Get-SPContentDatabase | %{Write-Output "- $($_.Name)”; foreach($site in $_.sites){write-Output $site.url}} > C:sitecollections.txt

This would create a .txt file with the same information:

Get Site Collections per Content Database oneliner 2

That’s all there is to it!

Get SharePoint lists by using PowerShell

Summary: This post describes how to get SharePoint lists by using PowerShell.

My last article explained how you can retrieve all your workflows from your SharePoint 2010 farm. After this, I got some requests from people that wanted an overview of all their used lists in SharePoint 2010.

For this, I adjusted my last script a little to output all your SharePoint lists and libraries. Or you can specify the -URL parameter to get just the lists from the particular site.

param
(
[string] $URL,
[boolean] $WriteToFile = $true
)

#Get all lists in farm
Add-PSSnapin Microsoft.SharePoint.PowerShell -ErrorAction SilentlyContinue

#Counter variables
$webcount = 0
$listcount = 0

if($WriteToFile -eq $true)
{
$outputPath = Read-Host "Outputpath (e.g. C:directoryfilename.txt)"
}
if(!$URL)
{
#Grab all webs
$webs = (Get-SPSite -limit all | Get-SPWeb -Limit all -ErrorAction SilentlyContinue)
}
else
{
$webs = Get-SPWeb $URL
}
if($webs.count -ge 1 -OR $webs.count -eq $null)
{
foreach($web in $webs)
{
#Grab all lists in the current web
$lists = $web.Lists
Write-Host "Website"$web.url -ForegroundColor Green
if($WriteToFile -eq $true){Add-Content -Path $outputPath -Value "Website $($web.url)"}
foreach($list in $lists)
{
$listcount +=1
Write-Host " – "$list.Title
if($WriteToFile -eq $true){Add-Content -Path $outputPath -Value " – $($list.Title)"}
}
$webcount +=1
$web.Dispose()
}
#Show total counter for checked webs & lists
Write-Host "Amount of webs checked:"$webcount
Write-Host "Amount of lists:"$listcount
}
else
{
Write-Host "No webs retrieved, please check your permissions" -ForegroundColor Red -BackgroundColor Black
}

You can run the script by running the .ps1 file. There are 2 parameters that you can add.

WriteToFile: Default is true. If the parameter is set to true, the console will ask for a output location where the text file will be saved.
URL: Default is empty. If the URL Parameter is set, only the web you specified and all webs under this web will be reported.

Here is a possible outcome:

Get SharePoint lists by using PowerShell

Creating an error in SharePoint 2010

Every once in a while, I give a SharePoint 2010 training, where the main focus lies in the infrastructural background.
One of these aspects is the monitoring of SharePoint 2010 and how to manage it and troubleshoot the best way possible. To do this, I use the ULSViewer.

You would think that it is pretty easy to create an error, seeing as how often you get an “unexpected error” in SharePoint. To make things easier, I found this article by Todd Carter. He shows a way to create custom error pages.

This gave me the idea to use the bug he pointed out, to demonstrate the ULS viewer.
From here it is pretty straightforward.
I opened the ULS viewer to start scanning my ULS by pressing CTRL + U. The ULS Viewer will automatically find the location of the logs and will open it real-time.

Using the bug, I navigated to “http://yourdomain.com/_layouts/Error.aspx?ErrorText=Error for demo purposes.”. I have registered this domain in my local DNS file for demonstrating purposes.
This shows the following screen:

image

Using the ULS viewer, i filtered the log for correlation ID “fb134e8c-553b-47ff-9a5d-e81d329e10dd “.

Now, the log has been filtered only to see the error I had created!

image

Of course, in this case, the only message that is visible is the GET request for the error page, but it did the trick!

Export/Import SharePoint 2010 solutions using PowerShell (Part 2/3)

Following up on the exporting of all solutions deployed in SharePoint 2010 in part 1, now it’s time to import these solutions to a different SharePoint 2010 farm.

Part 2 – Import all exported solutions to any SharePoint 2010 environment

In this part, we will cover how to add the solutions to another SharePoint 2010 farm. This is equal to using the “Add-SPSolution” command in PowerShell, only it does this for all solutions in a given directory.

Here is the script we will use:

#This script will add all solutions in the selected folder
Add-PSSnapin Microsoft.SharePoint.PowerShell

#Functions
function getPath
{
    #Get the directory that contains the solutions
    $inputPath = Read-Host "Enter the full path that contains the solutions (eg. C:solutions)"
    return $inputPath
}

function addSolutions($importPath)
{
$counterItemsAdded = 0
Write-Host "Adding the solutions to the SharePoint environment" -fore green

#Check if specified path exists, if it does, add all solutions in the directory
if((Test-Path $importPath) -eq $True)
    {
    $items = Get-ChildItem $importPath | ?{$_.extension -eq ".cab" -or $_.extension -eq ".wsp"} #Gets all files that have the .cab or .wsp extension
   #The following line checks all currently installed solutions and adds these to $solutions
    $solutions = (Get-SPFarm).Solutions | Foreach-Object{$solnames += $_.name;} #Get all current SPSolutions
   
        foreach($item in $items)
        {
        #If the solution does not exist within the current environment
        if($solnames -notmatch $item.toString())
            {
                Add-SPSolution -LiteralPath $item.Fullname | out-null
                Write-Host "Imported $item" -fore green
                $counterItemsAdded += 1
            }
        else
            {       
                Write-Host "$item already exists on the SharePoint environment." -fore red
            }
        }
        #If at least 1 solution has been added
        if($counterItemsAdded -ge 1)
        {
        Write-Host "The solutions have been added`n" -fore green
        Write-Host "Remember, the solutions will not be available until you deploy them" -fore yellow
        }
        else
        {
        Write-Host "The specified directory didn’t contain any valid .WSP or .CAB files" -fore red
        }
    }
#If the current path does not exist, ask for the path again   
else   
    {
    Write-Host "Directory does not exist. `nPlease enter the path in the following format: C:Solutions" -fore yellow
    addSolutions(getPath)
    }
}

Write-Host "This script will add all solutions in the specified directory to SharePoint 2010" -fore green

#Call functions
addSolutions(getPath)

The output of this script (when there are actually some valid .wsp files in the directory) will look something like this:

image

Like the output says, in order to use the solutions, you will have to deploy them first. You can do this manually, by using the PowerShell command “Install-SPSolution”, or you can go to part 3, where I created a script to do deploy ALL added solutions to your web applications.