Archive

Archive for the ‘PowerShell’ Category

Script to recrawl all crawl errors

I’m currently working on a document management project, which holds 10+ million documents. During a full crawl we had a temporary network issue, which resulted in 340.000 crawl errors. I didn’t want to do a new full crawl again, since the full crawl did finish with all documents. Instead, I want those items to be picked up in the next incremental crawl. Using Central Administration you can select the option “Recrawl the item in the next crawl” for each item which caused an error, but I obviously didn’t want to manually select this option for all errors.

To automate this, I’ve created a PowerShell script which can list the errors, but can also mark all errors automatically for the recrawl. The explanation of the script can be found in the comments of the script.

#——————————————————————————
# Provide parameters
#——————————————————————————
param (
   # Name of the search service application is mandatory
   [string] $SearchServiceApplicationName = $(throw “Please specify a search service application”),
  # By default, use all available content sources
   [string] $ContentSourceName = “”,
   # By default only a list of the errors is shown
   [switch] $RecrawlErrors = $false
)

#——————————————————————————
# Ensure the SharePoint PowerShell Snapin is loaded
#——————————————————————————
if ((Get-PSSnapin “Microsoft.SharePoint.PowerShell” -ErrorAction SilentlyContinue) -eq $null) {
    Add-PSSnapin “Microsoft.SharePoint.PowerShell”
}

#——————————————————————————
# Set some constant values
#——————————————————————————
# The id of the error stating a document will be processed in the next crawl
[int] $errorIdRetryNextCrawl = 437
# The number of documents which should be retrieved per batch from the ssa
[int] $batchSize = 1000
# 2 stands for Errors
[int] $errorLevel = 2

#——————————————————————————
# Retrieve the seach service application and crawl log
#——————————————————————————
$ssa = Get-SPEnterpriseSearchServiceApplication -Identity $SearchServiceApplicationName
$crawlLog = New-Object Microsoft.Office.Server.Search.Administration.CrawlLog $ssa

#——————————————————————————
# Retrieve the content source for which the errors should be loaded
#——————————————————————————
# Default use all content sources
[int] $contentSourceId = -1

# If a content source is provided, determine the ID
if([string]::IsNullOrEmpty($ContentSourceName) -eq $false) {
    write-host “Retrieving content source with the name $ContentSourceName… ” -NoNewline
    $contentSource = Get-SPEnterpriseSearchCrawlContentSource -SearchApplication $ssa -Identity $ContentSourceName -ErrorAction SilentlyContinue

    if($contentSource -eq $null) {
       write-host “Invalid content source provided” -ForegroundColor Red
      return
   }
    else {
       $contentSourceId = $contentSource.Id
write-host “Content source found” -ForegroundColor Green
    }
}
else {
   write-host “No content source provided, all available content sources will be used”-ForegroundColor Yellow
}

#——————————————————————————
# Process the crawl errors per error type
#——————————————————————————
write-host “”
write-host “Checking errors from the crawl log”
$crawlLog.GetCrawlErrors($contentSourceId, 1) | ForEach-Object {
    write-host ([string]::Format(“- {0}: {1}”, $_.ErrorCount, $_.ErrorMessage ))

    # Enable recrawl of errors for all errors except the recrawl on next crawl error
    if($RecrawlErrors -and $_.ErrorID -ne $errorIdRetryNextCrawl) {
       write-host “`t- Marking the errors for recrawl on next crawl”
# Get the first batch
       $processedItems = 0
       $errors = $crawlLog.GetCrawledUrls($false, $batchSize, “”, $true, $contentSourceId, $errorLevel, $_.ErrorID, [datetime]::MinValue,[datetime]::MaxValue)

       DO{
          write-host ([string]::Format(“`t`t – Processing batch {0}/{1}… “, $processedItems, $processedItems + $batchSize)) -NoNewline
   
# Recrawl the errors
          $errors | ForEach-Object {
          $crawlLog.RecrawlDocument($_.FullUrl) | Out-Null
       }
       write-host “done” -ForegroundColor Green
       $processedItems += $batchSize

       # Get the next batch
   $errors = $crawlLog.GetCrawledUrls($false, $batchSize, “”, $true, $contentSourceId, $errorLevel, $_.ErrorID, [datetime]::MinValue,[datetime]::MaxValue)

    }
    while ($errors -ne $null)
       write-host “”
    }
}

Advertisements

Forcefully delete site collection

August 21, 2014 8 comments

Today I found a site collection on a customer environment which gave a completely blank page when you opened it via a browser. It didn’t gave a 404 (Not Found) error, it was just a blank page. I decided to figure out what was happening and found that during the creation of the site collection, an IISRESET had taken place. Because of this, the site wasn’t completely provisioned. Well, if it wasn’t completely provisioned, I don’t need it… Nobody could have added content.

I found out that I couldn’t remove the site using Central Administration. When you navigate to the site collection using the “Delete a site collection” page, the details (right hand site of the page) where not loaded and you cannot select the site collection. So… I wanted to delete the site using PowerShell, but this gave me an error:

PS C:\Users\macaw> remove-spsite http://dms/case/P68430
Confirm
Are you sure you want to perform this action?
Performing the operation “Remove-SPSite” on target “http://dms/case/P68430“.
[Y] Yes [A] Yes to All [N] No [L] No to All [S] Suspend [?] Help (default is “Y”): Y
remove-spsite : <nativehr>0x80070003</nativehr><nativestack></nativestack>
At line:1 char:1
+ remove-spsite http://dms/case/P68430
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidData: (Microsoft.Share…mdletRemoveSite:SPCmdletRemoveSite) [Remove-SPSite], DirectoryNotFoundException
+ FullyQualifiedErrorId : Microsoft.SharePoint.PowerShell.SPCmdletRemoveSite

Apparently, the normal remove-spsite cmdlet cannot delete a site collection which is not fully provisioned, and this cmdlet doesn’t have a force flag. To forcefully delete the site collection, I used the SPContentDatabase.ForceDeleteSite method:

$siteUrl =http://dms/case/P68430
$site = get-spsite $siteUrl
$siteId = $site.Id
$siteDatabase = $site.ContentDatabase
$siteDatabase.ForceDeleteSite($siteId, $false, $false)

Create lookup field using PowerShell and CSOM

May 19, 2014 3 comments

For our projects we always try to avoid manual configurations. This is because it is a tedious and error prone process if you work with a DTAP environment. To avoid this, we also try to script as much as possible for SharePoint Online projects. Lately we worked with creating lookup fields in SharePoint online, using PowerShell and CSOM. Creating fields this way is pretty easy, but connecting lookup fields forced us to think about casting the Microsoft.SharePoint.Client.Field object to a Microsoft.SharePoint.Client.FieldLookup object.

Within CSOM this can be done by leveraging the ClientRuntimeContext.CastTo method, but… This is a generic method (object of type T). This is something which is not easily supported by PowerShell. To use this method, you can use reflection using the MakeGenericMethod method.

The full PowerShell script is provided below

#————————————————————-
# LOAD CLIENT ASSEMBLIES
#————————————————————-
$clientAssembliesFolder = “D:\ClientAssemblies”
Add-Type -Path (Join-Path -Path $clientAssembliesFolder -ChildPath “Microsoft.SharePoint.Client.dll”)
Add-Type -Path (Join-Path -Path $clientAssembliesFolder -ChildPath “Microsoft.SharePoint.Client.Runtime.dll”)

#————————————————————-
# INITIALIZE CONTEXT
#————————————————————-
[string]$siteUrl = "https://[UseYourOwn].sharepoint.com/sites/Dev"
[string]$username = “admin@[UseYourOwn].onmicrosoft.com”
[string]$password = “[UseYourOwn]”
$pwd = $password | ConvertTo-SecureString -AsPlainText -Force
$context = New-Object Microsoft.SharePoint.Client.ClientContext($siteUrl)
$credentials = New-Object Microsoft.SharePoint.Client.SharePointOnlineCredentials($username, $pwd)
$context.Credentials = $credentials

#————————————————————-
# LOAD CASTTO FOR LOOKUPS
#————————————————————-
$castToMethodGeneric = [Microsoft.SharePoint.Client.ClientContext].GetMethod(“CastTo”)
$castToMethodLookup = $castToMethodGeneric.MakeGenericMethod([Microsoft.SharePoint.Client.FieldLookup])

#————————————————————-
# LOAD LISTS
#————————————————————-
[string] $originaListTitle = “List1”
[string] $destinationListTitle = “List2”
$listOriginal = $context.Web.Lists.GetByTitle($originaListTitle)
$context.Load($listOriginal)
$listDestination = $context.Web.Lists.GetByTitle($destinationListTitle)
$context.Load($listDestination)
$context.ExecuteQuery() # This loads the necessary list ID

#————————————————————-
# CREATE LOOKUP
#————————————————————-
[string] $internalName = “LookupWithStaticName”
[string] $displayName = “LookupTest”
[string] $displayFieldForLookup = “Title”
[string] $lookupFieldXML = “<Field DisplayName=`”$internalName`” Type=`”Lookup`” />”
$option = [Microsoft.SharePoint.Client.AddFieldOptions]::AddFieldToDefaultView

$newLookupField
= $listDestination.Fields.AddFieldAsXml($lookupFieldXML, $true, $option)
$context.Load($newLookupField)
$lookupField = $castToMethodLookup.Invoke($context, $newLookupField)
$lookupField.Title = $displayName
$lookupField.LookupList = $listOriginal.Id
$lookupField.LookupField = $displayFieldForLookup
$lookupField.Update()
$context.ExecuteQuery()

SharePoint 2013 warm-up script

For SharePoint On Premise platforms it’s a good practice to use a warm-up script to avoid long loading times in the morning. By default IIS recycles the web application pools every night to clean up the memory and this is a good practice. Todd Klindt written a nice post about using the Invoke-WebRequest cmdlet which is available in PowerShell v3 and how to use this as basis for your warm-up script.

I used it as a basis and created the script you find below. Important notes:

  • The script will load the start page of the root site collection of every web application.
  • Different types of web templates, use different assemblies. If you want to preload all assemblies, ensure you load the different types of sites. The additionalUrls array is used for that in the script.
  • When you use multiple front-end servers, you want schedule the script on all front-end servers. Also make sure the server doesn’t use a load balancer when you are on the server itself, you can do this by updating the hosts file.

#——————————————————
# Ensure the SharePoint Snappin has been loaded
#——————————————————
if ( (Get-PSSnapin -Name “Microsoft.SharePoint.PowerShell” -ErrorAction SilentlyContinue) -eq $null ) {
    Add-PSSnapin “Microsoft.SharePoint.PowerShell”
}

#——————————————————

# Simple method to write status code with a colour
#——————————————————
function Write-Status([Microsoft.PowerShell.Commands.WebResponseObject] $response) {
    $foregroundColor = “DarkRed”
    if($response.StatusCode -eq 200) {
        $foregroundColor = “DarkGreen”
    
}
    write-host ([string]::Format(“{0} (Status code: {1})”, $response.StatusDescription, $response.StatusCode)) -ForegroundColor $foregroundColor
}

#——————————————————
# Warm-up all web applications
#——————————————————
Get-SPWebApplication | ForEach-Object {
    
write-host ([string]::Format(“WebApplication request fired for {0} [{1}]… “, $_.DisplayName, $_.Url)) -NoNewline
    
Write-Status -response (Invoke-WebRequest $_.url -UseDefaultCredentials -UseBasicParsing)
}

#——————————————————
# Since the root of web applications use different templates then other site collections, also load other sites of different
# types. This ensures their assemblies also get loaded in memory
#——————————————————
$additionalUrls = @(http://developmentserver/sites/search&#8221;,
 http://developmentserver/site/teamsite&#8221;)
$additionalUrls | ForEach-Object {
    write-host ([string]::Format(“Additional web request fired for Url: {0}… “, $_)) -NoNewline
    
Write-Status -response (Invoke-WebRequest $_ -UseDefaultCredentials -UseBasicParsing)
}

 

 

Re-activating web features within web application

One of my projects is a huge SharePoint 2013 On-Premise platform with 200.000+ (sub) sites. I’ve created a custom web template to ensure all sites are created the same way, with the same settings. A web template works very well for these environments, but when you update the template, the changes will not be made in all existing sites. The web template will only be applied when creating new sites.

I will not throw away all sites when we have new updates to re-create the sites, but I will re-active certain features to ensure the updates are applied.

The script I’m using is as followed:

#———————————————————————————————————————
# Add SharePoint PowerShell Snapin 
#———————————————————————————————————————
if ( (Get-PSSnapin -Name Microsoft.SharePoint.PowerShell -ErrorAction SilentlyContinue) -eq $null ) {
    Add-PSSnapin Microsoft.SharePoint.Powershell

#———————————————————————————————————————
# Set variables 
#———————————————————————————————————————
$webApplicationUrl = http://veemssdev02&#8221;
$featureIds = @(“e4acfa03-b1e6-4eed-aeab-1bd17551aa59”,“Macaw.SP2013.Intranet.InSite_AddDefaultPage_Web”

#———————————————————————————————————————
# Reactivate features 
#———————————————————————————————————————
Get-SPWebApplication -Identity $webApplicationUrl | get-spsite -Limit all | get-spweb -Limit all | ForEach-Object {
    write-host ([string]::Format(“Testing web {0} [{1}]”, $_.Title, $_.Url))

    foreach($featureId in $featureIds) {
        $feature = $_.Features | where {$_.DefinitionId -eq $featureId -or $_.Definition.DisplayName -eq $featureId}
        if($feature -ne $null) {
            write-host ([string]::Format(“`t- Feature {0} ({1}) found. Re-enabling the feature.”, $feature.Definition.DisplayName, $feature.DefinitionId))
           write-host “`t`t- Disabling feature”
            Disable-SPFeature -Identity $featureId -Url $_.Url -Confirm:$false
            write-host “`t`t- Enabling feature”
            Enable-SPFeature -Identity $featureId -Url $_.Url -Confirm:$false -force
        
}
    
}
}

When you do not want to re-activate features, but want to enable new features, you can simply use the same code, but remove the feature check (if($feature
-ne $null)
) and the Disable-SPFeature.

Categories: PowerShell, SharePoint

Generate overview of active features

For now the last section about generating documentation for your custom SharePoint solution. The following PowerShell scripts will provide you with an overview of activated features on your Farm, WebApplication and Site Collections. Since this script is meant for custom solutions, I assume you have used a prefix (Visual Studio adds the solution name by default as a prefix to all features).

Activated farm features

#———————————————————————————————————————-
# Add SharePoint PowerShell Snapin 
#———————————————————————————————————————-
if ( (Get-PSSnapin -Name Microsoft.SharePoint.PowerShell -ErrorAction SilentlyContinue) -eq $null ) {
    Add-PSSnapin Microsoft.SharePoint.Powershell


#———————————————————————————————————————-

# Get variables
#———————————————————————————————————————-
$featurePrefix = “Macaw”

#———————————————————————————————————————-
# Get information
#———————————————————————————————————————-
Get-SPFeature | where {$_.Scope -eq “Farm” -and $_.DisplayName.StartsWith($featurePrefix ,“CurrentCultureIgnoreCase”)} | Select DisplayName

Activated web application features

#———————————————————————————————————————-
# Add SharePoint PowerShell Snapin 
#———————————————————————————————————————-
if ( (Get-PSSnapin -Name Microsoft.SharePoint.PowerShell -ErrorAction SilentlyContinue) -eq $null ) {
    Add-PSSnapin Microsoft.SharePoint.Powershell

#———————————————————————————————————————-
# Get variables
#———————————————————————————————————————-
$featurePrefix = “Macaw”

#———————————————————————————————————————-
# Get information
#———————————————————————————————————————-
Get-SPWebApplication | foreach {
    Write-Host ([string]::Format(“Web Application: {0} ({1})”, $_.DisplayName, $_.Url))
   $_.Features | where {$_.Definition.DisplayName.StartsWith($featurePrefix ,“CurrentCultureIgnoreCase”)} | foreach {
        Write-Host ([string]::Format(“- {0} ({1})”, $_.Definition.DisplayName, $_.Definition.Id))
    }

    Write-Host “”
}

Activated site collection features

This script skips all personal sites, which you can easily add to your overview by removing the where clause.

#———————————————————————————————————————-
# Add SharePoint PowerShell Snapin 
#———————————————————————————————————————-
if ( (Get-PSSnapin -Name Microsoft.SharePoint.PowerShell -ErrorAction SilentlyContinue) -eq $null ) {
    Add-PSSnapin Microsoft.SharePoint.Powershell

#———————————————————————————————————————-
# Get variables
#———————————————————————————————————————-
$featurePrefix = “Macaw”

#———————————————————————————————————————-
# Get information
#———————————————————————————————————————-
Get-SPWebApplication | Get-SPSite | Where { $_.Url -notmatch “/personal/”} | foreach {
    Write-Host ([string]::Format(“Site Collection: {0} ({1})”, $_.RootWeb.Title, $_.Url))
   $_.Features | where {$_.Definition.DisplayName.StartsWith($featurePrefix ,“CurrentCultureIgnoreCase”)} | foreach {
        Write-Host ([string]::Format(“- {0} ({1})”, $_.Definition.DisplayName, $_.Definition.Id))
    }

    Write-Host “”
}

Previous posts about automatically generate documentation of your custom solution:

Generate solution documentation

April 3, 2014 1 comment

Sometimes you want an overview of your custom farm solutions which are deployed on your farm. Normally I prefix all my solutions to make it clear who created them. Using this prefix you can easily run a PowerShell script which provides you with a list of installed solutions and the web applications to which these solutions are deployed:

#———————————————————————————————————————-
# Add SharePoint PowerShell Snapin 
#———————————————————————————————————————-
if ( (Get-PSSnapin -Name Microsoft.SharePoint.PowerShell -ErrorAction SilentlyContinue) -eq $null ) {
    Add-PSSnapin Microsoft.SharePoint.Powershell

#———————————————————————————————————————-
# Get variables
#———————————————————————————————————————-
$solutionPrefix = “Macaw”

#———————————————————————————————————————-
# Get information
#———————————————————————————————————————-
foreach($solution in Get-SPSolution | Where {$_.Name.StartsWith($solutionPrefix ,“CurrentCultureIgnoreCase”)}) {
    Write-Host “Name: “ $solution.Name
    Write-Host “Identity: “ $solution.Id

    if($solution.get_ContainsWebApplicationResource() -eq $false) {
        Write-Host “Applications: Globally deployed”
    }
    else {
        Write-Host “Deployed to application(s):”
        $solution.DeployedWebApplications | foreach-object {
            Write-Host ([string]::Format(“- {0} ({1})”, $_.DisplayName, $_.Url))
        }
    }

    Write-Host “”
}

For a PowerShell script which documents all features, look at this post.

Ben Prins

What I want to remember about SharePoint

blog.frederique.harmsze.nl

my world of work and user experiences

Bram de Jager - Coder, Speaker, Author

Office 365, SharePoint and Azure