Cloud Hybrid Search proxy settings

The new Cloud Hybrid Search has been released in preview in September 2015. Seeing the demos I was really excited to try this. One of our customers was already in need of this technology, so we proposed to configure Cloud Hybrid Search on their test environment.

The onboarding process seemed pretty straightforward and we used http://blogs.msdn.com/b/spses/archive/2015/09/15/cloud-hybrid-search-service-application.aspx as a reference. We read the documentation, grabbed the on-boarding script, installed the prerequisites and got started.

After running the onboarding script the first time, some errors were thrown, but after running the script for the 2nd time, everything seemed to work out just perfectly. All steps were executed, no red parts in the output of the script.Cloud Hybrid Search proxy settingsThe next step would be to start a Full crawl. We created a specific content source in the on-premises environment. This content source contains only 1 site collection with a few documents inside, just to check the functionality. Unfortunately, the crawl failed. After inspecting the log, we saw the following:

AzureServiceProxy caught Exception: *** Microsoft.Office.Server.Search.AzureSearchService.AzureException: AzurePlugin was not able to get Tenant Info from configuration server
at Microsoft.Office.Server.Search.AzureSearchService.AzureServiceProxy.GetAzureTenantInfo(String portalURL, String realm, String& returnPropertyValue, String propertyName)
at Microsoft.Office.Server.Search.AzureSearchService.AzureServiceProxy.SubmitDocuments(String azureServiceLocation, String authRealm, String SPOServiceTenantID, String SearchContentService_ContentFarmId, String portalURL, String testId, String encryptionCert, Boolean allowUnencryptedSubmit, sSubmitDocument[] documents, sDocumentResult[]& results, sAzureRequestInfo& RequestInfo) ***

For some reason, SharePoint wasn’t able to submit index data for documents to Azure for processing. We created a thread on the TechNet forum for Cloud Search Service Application Preview.

As we were aware that a proxy was being used, we started checking the proxy configuration. There are a lot of places where you can configure a proxy if you are looking for them:

  • Web.config
  • Netsh winhttp set proxy
  • Browser settings for service account
  • Search Service Application proxy

After trying all these, we still couldn’t get it to work. After using Wireshark, we found out that the upload was still not using our proxy server. Instead, it tried uploading directly to Azure.

The solution
After discussing this with my colleague @vanHooijdonk, he reminded me that there is another place where you can configure proxy settings. The machine config: “C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Config\machine.config”

<system.net>
  <defaultProxy>
    <proxy usesystemdefault="false" proxyaddress="http://10.1.10.1:8080" bypassonlocal="true" />
  </defaultProxy>
</system.net>

After applying the proxy there, everything started to work! Place the proxy settings anywhere between the <configuration> and </configuration> tag. Personally, I placed it right before the end, so I can find it easier next time.

Please read my full blog post on how to configure Cloud Hybrid Search here: https://www.sharepointrelated.com/2015/12/21/everything-you-need-to-know-about-cloud-hybrid-search/

Thanks to @johankroese and @vanHooijdonk for helping with troubleshooting this issue!

Office 365 Video: My first impression

I’ve had the opportunity to look into the possibilities for organizations to use Office 365 Video. Using videos within an organization can be a great way to connect to your employees. This could be a welcome video from the new CEO, a safety regulation video or training material to help new employees get on their way.

Homepage
The Video portal homepage consists of a few key sections. The Spotlight area gives you the possibility to show videos, which have a strong presence on the homepage. The Popular videos section shows videos that are popular within your organization. The last section is the Spotlight channels. Here you can select 3 channels, from which you show the spotlight videos on your homepage.

Office 365 Video portal settings
Office 365 Video portal settings

The popular videos in particular is a great way to see which videos are popular in your organization. The only downside is that an administrator cannot remove or influence the popular video section.
For each video on the homepage you can see the total views and the length of the video.

Personally, I would like to have a little more control over the Office 365 Video homepage. For instance, a channel overview on the left or right side of the screen would be useful to navigate between channels. The possibility to view a video from the homepage could be easier as well, as now you have to click the context action menu (the 3 dots). A pop-up will be shown, starting the video.

Channels
Besides showing videos on your homepage, an administrator has the possibility to create channels. Permissions can be set per channel to allow only certain people (or groups) to view the videos in the channel.

When an administrator creates a channel, a SharePoint Online site collection is created in the background. This is where the video and all metadata will be saved. A copy will be sent to Azure Media services. Microsoft posted this nice video on YouTube explaining how this works behind the scene.

Start a new channel
Start a new channel

The channel creation process could be more smoothly, as the administrator is required to wait for the site collection to be created, which takes some time. When you close the window while a channel is creating, the channel will still be created, but the channel color you selected won’t be saved. It would be nice if the creation could happen behind the screens, so the administrator doesn’t have to wait for the site collection to be provisioned.

Video
While watching a video, Office 365 Video provides you with several options. You can share the video, see the description of the video, or even comment on the video by using the inline Yammer conversation on the right side of the screen. Below the video is a list of videos that you may also like.

Video Yammer conversations
Inline Yammer conversations

Videos will be processed and displayed in Delve as well. This is a really neat feature that helps employees engage with Office 365 Video without actually navigating to Office 365 Video in the first place, as relevant videos are being showed in Delve.

Conclusion
In conclusion: Office 365 Video is a great initiative that has a lot of potential. At the moment of this writing, I think the service could still be improved by integrating Office 365 Video with other Office 365 services even more. In particular, some of the features that SharePoint online has, could help make Office 365 Video even better. As Office 365 Video is still a relatively new service, I expect Microsoft will make regular improvements that will add business value organizations.

If you are currently testing or using Office 365 Video, make sure to provide feedback to help improve the service. On the top right corner of each page you will find the option to send Feedback to Microsoft.

feedback
Feedback

Change site collection URL in SharePoint 2013

Summary: This post describes the different ways to change the site collection URL in SharePoint 2013.

So, your manager/the business asks you to create a few site collections for some departments in your organization. You quickly spin up some site collections for them to use. After a few weeks, the business decides (of course) SharePoint is great, but the URL’s we chose weren’t all that great. Can you please change them?

There are alot of blog posts out there that describe the different possibilities in this scenario:
– Using backup and restore.
– A great article by Todd Klindt that tells you how to use the Copy-SPSite cmdlet to achieve the same goal but easier!

There is another (easier) way in some scenario’s in which you can change the site collection URL by just using 2 lines of PowerShell with only a second of waiting time!

The solution
Let’s get down to it.. I created this very nice Marketing site collection, using the url: http://sharepoint/sites/marketing. Now I would like to change this to http://sharepoint/sites/sales. Just use the following 2 lines of PowerShell code and you are done!

 $site = Get-SPSite http://sharepoint/sites/marketing
$site.Rename("http://sharepoint/sites/sales")

That is all.. try it out and see that your new URL is working, and the old URL is not working anymore!
*NOTE* Thanks to Jaymeson in the comments for pointing out an IISRESET is needed as well!

A little catch
There is a little catch to this. You can only use this to rename site collection URL’s that
– Use “Wildcard inclusion” Managed Paths.
– Are Host named site collections (In which case you could also use Set-SPSiteURL)

You can’t use it to change http://sharepoint/sites/marketing to http://sharepoint/marketing (Even if the Explicit inclusion managed path exists).

Hope this helps anyone out there!

Download content from a site collection

I’ve been working on a script that will allow you to download all files that are stored in SharePoint in a given site collection.

If the path does not exist, the script will prompt you to create it for you. Before the script runs, it also checks if the site collection exists.

Run the script like this:

.\Get-SPContent.ps1 -SiteCollection "<SiteCollectionURL>" -Destination "<Path>"

Download content

The console shows which libraries were exported to your file system.

—– * Advanced * —–

If you have specific requirements as to which (type of) libraries you want to export, you can change the following line to fit your requirements:

$lists = $web.lists | ?{$_.itemcount -ge &quot;1&quot; -And $_.Hidden -eq $false -And $_.BaseType -eq &quot;DocumentLibrary&quot;} #Excludes all hidden libraries and empty libraries

Below is the code you can save as Get-SPContent.ps1

param
(
[Parameter(Mandatory=$true)]
[ValidateScript({asnp *sh* -EA SilentlyContinue;if (Get-SPSite $_){$true}else{Throw &quot;Site collection $_ does not exist&quot;}})]
[string]$SiteCollection,
[Parameter(Mandatory=$true)]
[ValidateScript(
{
if (Test-Path $_)
{$true}
else{
$d = $_
$title = &quot;Create Folder?&quot;;
$message = &quot;$_ doesn't exist, do you want the script to create it?&quot;;
$yes = New-Object System.Management.Automation.Host.ChoiceDescription &quot;&amp;Yes&quot;, &quot;Creates directory $_&quot;;
$no = New-Object System.Management.Automation.Host.ChoiceDescription &quot;&amp;No&quot;, &quot;Exits script&quot;;
$options = [System.Management.Automation.Host.ChoiceDescription[]]($yes,$no);
$result = $host.ui.PromptForChoice($title,$message,$options,1);
switch($result)
{
0 {New-Item $d -Type Directory;$true}
1 {Throw &quot;Please create the folder before running the script again. `nExiting script&quot;}
}
}
})]
[string]$Destination
)

Asnp *sh* -EA SilentlyContinue

Start-SPAssignment -Global | Out-Null

function Get-SPWebs($SiteCollection){
$SiteCollection = Get-SPSite $SiteCollection
$webs = @()
$SiteCollection.allwebs | %{$webs += $_.url}
return $webs
}

function Get-SPFolders($webs)
{
foreach($web in $webs)
{
$web = Get-SPWeb $web
Write-Host &quot;`n$($web.url)&quot;

$lists = $web.lists | ?{$_.itemcount -ge &quot;1&quot; -And $_.Hidden -eq $false -And $_.BaseType -eq &quot;DocumentLibrary&quot;} #Excludes all hidden libraries and empty libraries
#$lists = $web.lists | ?{$_.title -eq &quot;Documents&quot; -and $_.itemcount -ge &quot;1&quot; -And $_.BaseType -eq &quot;DocumentLibrary&quot;} #Change any identifier here
foreach($list in $lists)
{
Write-Host &quot;- $($list.RootFolder.url)&quot;

#Download files in root folder
$rootfolder = $web.GetFolder($list.RootFolder.Url)
Download-SPContent($rootfolder)

#Download files in subfolders
foreach($folder in $list.folders)
{
$folder = $web.GetFolder($folder.url)
Download-SPContent($folder)

}

}
$web.dispose()
}
}

function Download-SPContent($folder)
{
foreach($file in $folder.Files)
{
$binary = $file.OpenBinary()
$stream = New-Object System.IO.FileStream($destination + &quot;/&quot; + $file.Name), Create
$writer = New-Object System.IO.BinaryWriter($stream)
$writer.write($binary)
$stream.Close()
$writer.Close()
}
}

$webs = Get-SPWebs -SiteCollection $Sitecollection
Get-SPFolders -Webs $webs

Stop-SPAssignment -Global

Encrypting text-based files using PowerShell

 

This script will encrypt text-based files using PowerShell.

Please find the script in the TechNet Gallery here. All updates to this script will also be managed here: https://gallery.technet.microsoft.com/Encrypting-your-text-based-d939876b.

As a SharePoint administrator I am often asked to script certain tasks. Of course, PowerShell does a great job at this, as it allows me to automate repetitive tasks.

One commonly faced issue (for me), is that whenever I am using a XML file for those tasks, I sometimes have to include a password in the XML file for some scripts to run. An example is AutoSPInstaller. Personally, I don’t think you should have your passwords in an unencrypted XML file just sitting around for some developer to find and possibly abuse it. To work around this, I created some scripts to help me address this issue.

 

 

Add-PSSnapIn Microsoft.SharePoint.PowerShell shortcut

Are you tired of typing Add-PSSnapIn Microsoft.SharePoint.PowerShell every time you open your PowerShell console?
You can do this by Adding/Editing profile.ps1 (I don’t do this because I have too many machines where I should change this)

If you find you do not want to change the profile.ps1 on every server you are working on, you can type this:

asnp *sh*

This adds all SnapIns that contain *sh*. In most cases, this will only add the Microsoft.SharePoint.PowerShell SnapIn.
Credits go to Koen Zomers!

Happy PowerShelling!

 

Restore deleted site collections SharePoint 2013

In SharePoint 2013 it is possible to restore a deleted site collection. For more information, read this article: http://technet.microsoft.com/en-us/library/hh272537.aspx

You can use the Restore-SPDeletedSite cmdlet to restore a site collection.

However, if you removed the site collection using the Remove-SPSite cmdlet using PowerShell, the site collection will not be stored in a SPDeletedSite object.

This means you cannot restore a site collection that has been removed using PowerShell.

 

Add PDF mimetype for all Web Applications oneliner

By default, PDF files cannot be opened directly from SharePoint 2010/SharePoint 2013.

To add the PDF mimetype to all Web Applications (Instead of doing it seperately for each Web Application), you can use the following oneliner:

Get-SPWebApplication | %{$_.AllowedInlineDownloadedMimeTypes.Add("application/pdf");$_.Update()}

Get all subsites of a subsite using PowerShell

Getting a list of all subsites of a particular site (not a site collection) was a little more work than I expected, so here is how I did it.

Let’s say we have the following situation site structure:

SiteStructure

What if we want an overview of all sites under “Https://portal.sharepointrelated.com/Projects”?

My first thought was to use the “Webs” property of the SPWeb object. Unfortunately, this only shows the direct subsites for this site. This means that for “Https://portal.sharepointrelated.com/projects”, it only shows the Level 3 sites.

Solution

To work around this, I used the “AllWebs” property of the SPSite object and filtered the URL’s starting with “Https://portal.sharepointrelated.com/projects”.

Here is the code used: (Download .zip file)


param ( [Parameter(Mandatory=$true)][ValidateNotNullOrEmpty()] [String]$StartWeb, [Boolean]$IncludeStartWeb = $true )

Add-PSSnapin Microsoft.SharePoint.PowerShell -ErrorAction SilentlyContinue

$subsites = ((Get-SPWeb $StartWeb).Site).allwebs | ?{$_.url -like "$StartWeb*"}

foreach($subsite in $subsites) { Write-Host $subsite.url }

As you can see in the source code, I added 2 parameters to the script:

StartWeb: String. This is the starting URL. All subsites under this site will be showed in the result.

IncludeStartWeb: Boolean. When set to $false, the output will not include the URL provided in the StartWeb parameter.

Remove users from User Information List (people picker) using PowerShell

This post will describe how you can remove users in bulk from the User Information List using a PowerShell script and a simple CSV file.

In SharePoint 2010, the people picker retrieves data from multiple sources.

– The Site Collection’s User Information List (UIL);
– Active Directory.

When you delete a user from Active Directory, this will not mean the user isn’t searchable in SharePoint. Actually, if you look for this person in the people picker, you will probably find him/her. As the data is pulled from different sources, there may be several causes for this.

Assuming the user is really deleted from the Active Directory, I will give you some pointers as to how to “delete” the users from the People picker.

Scenario
I have a user called Kim Akers in my Contoso environment. She has permissions on several sites/subsites, and placed documents and list items in multiple places.

Example site

She also has a MySite.

Profile

For some reason, Kim is fired. The Active Directory administrators remove her account from Active Directory.

However, when I look at the People Picker in SharePoint, I can still find this user.

People picker

Why is this happening?

Every user that is given direct permissions, or has logged in to SharePoint, will be added to the Site Collection’s User Information List. This is a hidden list, that you can access by going to your site collection’s URL and add /_catalogs/users/simple.aspx. For instance: http://portal.contoso.com/_catalogs/users/simple.aspx.

This will show a list of all users that have logged in on your SharePoint. Sure enough, Kim can still be found here, even though her account has been deleted in Active Directory.

User Information List

Solution

To remove the user from the information list, you can use the GUI. If you want more information on how to do this, read this article. Also, make sure the profile for this user is not in the Profile Database. You can remove users from the Profile Database directly by going to Central Administration -&> Application Management -&> Manage Service Application -&> Click your User Profile Service Application -&> Manage User Profiles -> Find profile by entering the name -> Select the name in the list, and click Delete.

In my case, I wanted to remove a list of users from All site collections, because I am certain that these users will never log in again, and I don’t want them to show in the people picker. The below script will do just that!

(
[Parameter(Mandatory=$true)][ValidateScript({Test-Path $_ -Include "*.csv"})]
[String]$CSVPath
)

#This script will remove users specified in the CSV.

$CSVFile = Import-CSV $CSVPath
Add-PSSnapin Microsoft.SharePoint.PowerShell -EA SilentlyContinue

#Get all site collections
$Sites = Get-SPSite -Limit All
$AllSites = @()
foreach($Line in $CSVFile)
{
foreach($Site in $Sites)
{
#Get the rootweb for the site collection
$RootWeb = $Site.RootWeb
If([bool](Get-SPUser $Line.Username -Web $RootWeb -EA SilentlyContinue) -eq $True)
{
#Remove the user from the User Information List
Remove-SPUser -Identity $Line.username -Web $RootWeb -Confirm:$False
$AllSites += $RootWeb.Url
}
}
if(!($AllSites).count -eq 0)
{
#Give feedback on deleted users
Write-Host "Removed user $($Line.username) from:" -Fore "Magenta"
foreach($S in $AllSites){Write-Host "- $S"}
Write-Host ""
$AllSites = @()
}
}

I save the above text in a .ps1 file called Remove-SPUserBulk.ps1.

Next, I create a CSV file (Users.csv) that will contain all the users that I want to remove. My demo CSV looks like this:

CSV

As you can see, I added a non-existing account, to show that the script actually just deletes the existing user, and the output is correct. I run the script by going to the location where the Remove-SPUserBulk.ps1 file is located, and enter: “Remove-SPUSerBulk.ps1 -CSVPath “C:scriptsUsers.csv”.

Below is the result.

Script

Be aware that if the user is a site collection administrator, you will get an error stating you cannot delete the owners of a Web site collection.