Increase List items threshold in SharePoint

We think a lot if want to increase the list view items threshold for the web application because it will be applied to all the site collections that running under the same web application. When we work with list there are many question comes in our mind.

1. SharePoint list supports 3 million item in a list but if threshold limit is crossed the the user will not be able perform many operation including OOTB (Additional filter on list view, grouping, adding new filed or updating existing field). How to avoid such situation?

2. How to increase the threshold for a single list or single web(sub site)?

3. Are there any possibilities for viewing all items/ navigating to all items without any trouble?

4. How majorly system performance impacted if list view threshold increase for the total web application? Are there any best practices?

5. Why Server object model not returning all items from a view when fetching items from specific view?

6. Why OOTB view query is overriding custom CAML query when fetching specific view with CAML query?

and there are more question may arise if you dig further on SharePoint list.

But we can increase the List threshold for a specific list using server object mode / PowerShell scripts.

Server Object Model

using(SPSite site = new SPSite(@"http://kmsnet:5050"))
{
   foreach(SPWeb web in site.AllWebs)
   {
     web.AllowUnsafeUpdates=true;
     SPList lst=web.Lists.TryGetList("MyList");
     if(lst != null)
     {
        lst.EnableThrottling=false;
        lst.Update();       
     }
     web.Update();
     web.AllowUnsafeUpdates=false;
     web.Dispose();
   }
}

PowerShell

Add-PSSnapin Microsoft.SharePoint.PowerShell #-ErrorAction SilentlyContinue

$site = Get-SPSite -Identity "http://kmsnet:5050/"

foreach($web in $site.AllWebs)
{
   $web.AllowUnsafeUpdates = $True;
   $list=$web.Lists.TryGetList("MyList");
   if($list -ne $null)
   {
        $list.EnableThrottling = $False;
        $list.Update();
        Write-Host "List Updated in web" $web.Title
   }
   $web.Update();
   $web.AllowUnsafeUpdates = $False;
   $web.Dispose()
}
$site.Dispose();

Caching techniques in SharePoint

SharePoint Server 2010 provides three types of caches that help improve the speed at which Web pages load in the browser.

1.       The BLOB cache.

2.      The Page Output Cache.

3.       The Object Cache.

Page output and object cache require that you activate the ‘SharePoint Server Publishing’ feature located in the site features of your web site.

BLOB cache:

SharePoint Server 2010 provides a disk-based cache that stores files that are used by Web pages to help them load quickly in the browser, and reduces the load on the database server when it uses those files. The BLOB cache is stored directly on the hard disk drive of a front-end Web server computer. By default, the BLOB cache is off and must be enabled to use the functionality it provides. When you enable the BLOB cache on your front-end Web server, you reduce the load on the SharePoint Server 2010 database server created by read requests from Web browsers. You enable the BLOB cache in the Web.config file of the Web application to which you want to apply it. The changes that you make to the Web.config file will be applied to all site collections within the Web application. To configure and Setup BLOB Caching read here.

1. BLOB cache should be used when pages that are accessed frequently have JavaScript, CSS, images files, and large rich media files that can be cached on the WFEs
2. It is recommended that you not set the cache size smaller than 10 GB for BLOB Caching. When you set the cache size, make sure to specify a number large enough to provide a buffer at least 20 precent bigger than the estimated size of the content that will be stored in the cache.
3. For a publishing site for which most of the visitors are anonymous or where most of the files are static content, enable the BLOB cache for as many file types as possible.
4. For other sites that contain lots of media assets that are read-only, or where only a small percentage of the media assets are updated, enable the BLOB cache for media files only.
5. Make sure that you put the BLOB cache on a drive that has sufficient disk space available in front-end Web server which to store the cache.
6. Select a drive that will be used by as few processes as possible so that the BLOB cache process does not encounter conflicts when it tries to access the drive
7. If there is a high read to write ratio BLOB caching should be used. For instance you would want to cache a site logo that is used on every page request versus a collaboration word document that is actively updated.
8. BLOB caching is optimized for supporting large files which can significantly reduce bandwidth between the WFEs and SQL Server.
9. BLOB caching is optimized to support cache control headers so that clients can cache small files which can reduce overall number of hits to the WFEs.
10. If there is anonymous access, there can be dramatic improvements because permissions do not have to be validated for cache files.
11. Client applications that use range requests can optimize load times to access large files.

Before you enable the BLOB cache, carefully consider the scenario in which you plan to use it. If your site will be used for heavy collaboration, enabling the BLOB cache might temporarily affect the performance of your site while the files to be cached are first written to the disk. After the files have been stored in the cache, site performance will improve, so take this into consideration when you decide whether or not to enable the cache. Base your decision to enable BLOB caching on the following criteria:

·         For a publishing site for which most of the visitors are anonymous or where most of the files are static content, enable the BLOB cache for as many file types as possible.

·         For other sites that contain lots of media assets that are read-only, or where only a small percentage of the media assets are updated, enable the BLOB cache for media files only.

Page output cache profiles:

The second caching option you have with SharePoint 2010 is ASP.net Output Cache. This is an in-memory cache that saves rendered ASPX pages. Using Output cache improves performance in two ways first it reduces the amount of SQL calls. Second it reduces workload on the WFE because pages do not need to be re-rendered. Along those lines if the pages are anonymous, then no SQL check needs to be done at all present the cached pages. Microsoft testing concluded a ninefold improvement in throughput when compared to having to render the page every time it was rendered. To configure and Setup Output Caching read here.

1. The only catch for using Output Cache is that it can only be used in conjunction with publishing pages. It cannot be used with a collaboration site.
2. Some examples of rules that can be capture in a cache profile would be to not cache if the requestor is a user who can edit pages to ensure they see the latest version of content.
3. The cache profile also specifies rules for when a page is invalidated so that when the next request is made, it comes from the database.
4. One of the main considerations for Output cache is the memory needed to support it. For each rendered page, 2(size of the page) + 32KB is needed to store the rendered page in memory.
5. Output Cache should not be used with sites using a low read to write ratio because frequent changes to content make it hard to keep the cache fresh.
6. On multi-WFE, output caching may affect consistency.

 Object cache

Object cache is the third caching option we have for SharePoint 2010. What Object cache does is stores metadata about SharePoint Server objects (like SPWeb, SPSite, SPList, etc.) on the WFEs. When a page is rendered, if there is data that needs to be retrieved through these objects, the SQL Server will not be hit. Features of SharePoint that uses Object cache are publishing, content query web part, navigation, search query box and metadata navigation. These features are specifically written to use the Object cache API instead of the SharePoint API directly. Developers writing custom functionality can also tap into the Object cache API. To configure and Setup Output Caching read here.

1. The object cache cannot be disabled, and there is very little that can be configured. None of the out of box settings should have to be changed. Following article was written provides some additional information on this cache: Object Cache.
2. They are normal ASP.net caching and can be a problem when Server data changes in case of multiple farm.
3. It is recommended that you not set the cache size smaller than 3 MB.

Overall for caching we should keep in mind that any data where in Read/write ratio is less we should avoid caching. Only things which is static most of the time need to be cached like images etc.

When you flush the BLOB cache, you clear the contents of the BLOB cache for a Web application. This is useful if the BLOB cache becomes out of sync with the content. For example, after you restore a content database, the BLOB cache will be out of sync with the content. To correct that situation, you must flush the BLOB cache. The following procedure describes how to flush the BLOB cache for a Web application.

Important Information on Flushing:

a.       Flushing the BLOB cache for a Web application affects all site collections in the Web application.

b.      You cannot use the user interface to flush the BLOB cache. Instead, you use Windows PowerShell and the SharePoint object model to complete this task.

This is the link which tells you how to flush BLOB cache

Useful links

Configure Cache in SharePoint 2013

Custom Caching Overview

Output caching in SharePoint 2010

SharePoint 2010 developing for performance: caching-in-sharepoint-2010

Common coding issues in SharePoint caching

Machine caching in SharePoint 2010

Best coding practices

http://msdn.microsoft.com/en-us/library/aa661294.aspx

http://msdn.microsoft.com/en-us/library/aa661294.aspx

http://msdn.microsoft.com/en-us/library/ms550239.aspx

http://msdn.microsoft.com/en-us/library/aa622758.aspx

http://technet.microsoft.com/en-us/library/cc770229.aspx

http://technet.microsoft.com/en-us/library/cc770229.aspx#BLOB

http://www.zimmergren.net/archive/2008/10/07/web-part-caching-%E2%80%93-a-simple-approach.aspx

http://msdn.microsoft.com/en-us/library/bb687949%28office.12%29.aspx#UsingSPData

http://technet.microsoft.com/en-us/library/ee424404.aspx#Section1c

Change site master page programmatically in SharePoint

It very simple to change the site's master page to custom master page programmatically in SharePoint 2007 environment. Recently we had released a new custom master pages. The master pages are with left menu place holders and another one is without left menu.

Provided a utility to update all sub sites master page. Find the same source code below.

siteUrl=@"http://kmsnet:2020/";
using (SPSite site =new SPSite(siteUrl))
{
foreach(SPWeb web in site.AllWebs)
{
SPSecurity.RunWithElevatedPrivileges(delegate ()
{
if(web.CustomMasterUrl.ToLower().Equals(("/_catalogs/masterpage/NoLeftMenu.master").ToLower()))
{
web.CustomMasterUrl = "/_catalogs/masterpage/NewNoLeftMenu.master";
web.Update();
}
if (web.CustomMasterUrl.ToLower().Equals(("/_catalogs/masterpage/LeftMenu.master").ToLower()))
{
web.CustomMasterUrl = "/_catalogs/masterpage/NewLeftMenu.master";
web.Update();
}
});
}
}

Start SharePoint timer job and wait till job finished using PowerShell

Start or Run SharePoint timer job and wait till the job get completed.

Below PowerShell scripts to start the Document ID timer jobs(Document enable, Document Id Assignment) for specific web application. The function need to parameters one is job name and next one is web application.

Add-PSSnapin Microsoft.SharePoint.PowerShell -ErrorAction SilentlyContinue

function StartJobOnWebApp
{
    param([string]$WebAppName, [string]$JobName)

    Write-Host " ";
    Get-SPWebApplication;
    Write-Host " ";
 
    $WebApp = Get-SPWebApplication $WebAppName;
 

    ##Getting right job for right web application
    $job = Get-SPTimerJob | ?{$_.Name -match $JobName} | ?{$_.Parent -eq $WebApp}
    if($null -ne $job)
    {
        $startet = $job.LastRunTime
        Write-Host -ForegroundColor Yellow -NoNewLine "Running"$job.DisplayName"Timer Job."
        Start-SPTimerJob $job

        ##Waiting til job is finished
        while (($startet) -eq $job.LastRunTime)
        {
            Write-Host -NoNewLine -ForegroundColor Yellow "."
            Start-Sleep -Seconds 2
        }

        ##Checking for error messages, assuming there will be errormessage if job fails
        if($job.ErrorMessage)
        {
            Write-Host -ForegroundColor Red "Possible error in" $job.DisplayName "timer job:";
            Write-Host "LastRunTime:" $lastRun.Status;
            Write-Host "Errormessage:" $lastRun.EndTime;

        }
        else
        {
            Write-Host -ForegroundColor Green $job.DisplayName"Timer Job has completed.";
        }
    }
    else
    {
        Write-Host -ForegroundColor Red "ERROR: Timer job" $job.DisplayName "on web application" $WebApp "not found."
    }

}

# Input parameter for site collection/web application
$siteUrl = "http://kmsnet:2020/"

$rootSite = New-Object Microsoft.SharePoint.SPSite($siteUrl)

$spWebApp = $rootSite.WebApplication

#run document ID timer job
StartJobOnWebApp  $spWebApp.url "DocIdEnable"

# run Document ID Assignment  timer job
StartJobOnWebApp  $spWebApp.url "DocIdAssignment"

Find single user permission across differerent SharePoint scopes

PowerShell script which scans these areas to to retrieve a specific user's access rights:

·         Farm Administrator's Group

·         Central Administration Web Application Policies

·         Site Collection Administrators

·         Scans the all Site collections and Sub-sites with Unique Permissions

·         Scans all Lists and Libraries with unique permissions

·         Scans all Groups which has permissions on sites and Lists

After executing the script, it generates a CSV file (Tab Separated, In fact!) with details: URL, Site/List, Title, Permission Type, Permissions as in the below screenshot. 

Add-PSSnapin Microsoft.SharePoint.PowerShell -ErrorAction SilentlyContinue

Function GetUserAccessReport($WebAppURL, $SearchUser)

{

 #Get All Site Collections of the WebApp

 $SiteCollections = Get-SPSite -WebApplication $WebAppURL -Limit All

#Write CSV- TAB Separated File) Header

"URL `t Site/List `t Title `t PermissionType `t Permissions" | out-file UserAccessReport.csv

  #Check Whether the Search Users is a Farm Administrator

  $AdminWebApp= Get-SPwebapplication -includecentraladministration | where {$_.IsAdministrationWebApplication}

    $AdminSite = Get-SPweb($AdminWebApp.Url)

   $AdminGroupName = $AdminSite.AssociatedOwnerGroup

    $FarmAdminGroup = $AdminSite.SiteGroups[$AdminGroupName]

     foreach ($user in $FarmAdminGroup.users)

      {

       if($user.LoginName -eq $SearchUser)

    {

     "$($AdminWebApp.URL) `t Farm `t $($AdminSite.Title)`t Farm Administrator `t Farm Administrator" | Out-File UserAccessReport.csv -Append

    }    

      }

 #Check Web Application Policies

 $WebApp= Get-SPWebApplication $WebAppURL

 foreach ($Policy in $WebApp.Policies)

   {

   #Check if the search users is member of the group

  if($Policy.UserName -eq $SearchUser)

     {

    #Write-Host $Policy.UserName

     $PolicyRoles=@()

     foreach($Role in $Policy.PolicyRoleBindings)

    {

     $PolicyRoles+= $Role.Name +";"

    }

    #Write-Host "Permissions: " $PolicyRoles

    "$($AdminWebApp.URL) `t Web Application `t $($AdminSite.Title)`t  Web Application Policy `t $($PolicyRoles)" | Out-File UserAccessReport.csv -Append

   }

   }

  #Loop through all site collections

   foreach($Site in $SiteCollections)

    {

   #Check Whether the Search User is a Site Collection Administrator

   foreach($SiteCollAdmin in $Site.RootWeb.SiteAdministrators)

       {

    if($SiteCollAdmin.LoginName -eq $SearchUser)

   {

    "$($Site.RootWeb.Url) `t Site `t $($Site.RootWeb.Title)`t Site Collection Administrator `t Site Collection Administrator" | Out-File UserAccessReport.csv -Append

   }    

  }

    #Loop throuh all Sub Sites

       foreach($Web in $Site.AllWebs)

       {

   if($Web.HasUniqueRoleAssignments -eq $True)

             {

          #Get all the users granted permissions to the list

             foreach($WebRoleAssignment in $Web.RoleAssignments )

                 {

                   #Is it a User Account?

      if($WebRoleAssignment.Member.userlogin)  

       {

          #Is the current user is the user we search for?

          if($WebRoleAssignment.Member.LoginName -eq $SearchUser)

         {

          #Write-Host  $SearchUser has direct permissions to site $Web.Url

          #Get the Permissions assigned to user

           $WebUserPermissions=@()

             foreach ($RoleDefinition  in $WebRoleAssignment.RoleDefinitionBindings)

             {

                             $WebUserPermissions += $RoleDefinition.Name +";"

                            }

          #write-host "with these permissions: " $WebUserPermissions

          #Send the Data to Log file

          "$($Web.Url) `t Site `t $($Web.Title)`t Direct Permission `t $($WebUserPermissions)" | Out-File UserAccessReport.csv -Append

         }

       }

     #Its a SharePoint Group, So search inside the group and check if the user is member of that group

     else

      {

                        foreach($user in $WebRoleAssignment.member.users)

                            {

           #Check if the search users is member of the group

         if($user.LoginName -eq $SearchUser)

          {

           #Write-Host  "$SearchUser is Member of " $WebRoleAssignment.Member.Name "Group"

            #Get the Group's Permissions on site

         $WebGroupPermissions=@()

            foreach ($RoleDefinition  in $WebRoleAssignment.RoleDefinitionBindings)

            {

                           $WebGroupPermissions += $RoleDefinition.Name +";"

                           }

         #write-host "Group has these permissions: " $WebGroupPermissions

         #Send the Data to Log file

         "$($Web.Url) `t Site `t $($Web.Title)`t Member of $($WebRoleAssignment.Member.Name) Group `t $($WebGroupPermissions)" | Out-File UserAccessReport.csv -Append

        }

       }

      }

                    }

    }

    #********  Check Lists with Unique Permissions ********/

              foreach($List in $Web.lists)

              {

                  if($List.HasUniqueRoleAssignments -eq $True -and ($List.Hidden -eq $false))

                  {

                     #Get all the users granted permissions to the list

                foreach($ListRoleAssignment in $List.RoleAssignments )

                    {

                      #Is it a User Account?

         if($ListRoleAssignment.Member.userlogin)  

          {

             #Is the current user is the user we search for?

             if($ListRoleAssignment.Member.LoginName -eq $SearchUser)

            {

             #Write-Host  $SearchUser has direct permissions to List ($List.ParentWeb.Url)/($List.RootFolder.Url)

             #Get the Permissions assigned to user

              $ListUserPermissions=@()

                foreach ($RoleDefinition  in $ListRoleAssignment.RoleDefinitionBindings)

                {

                                $ListUserPermissions += $RoleDefinition.Name +";"

                               }

             #write-host "with these permissions: " $ListUserPermissions

             #Send the Data to Log file

             "$($List.ParentWeb.Url)/$($List.RootFolder.Url) `t List `t $($List.Title)`t Direct Permissions `t $($ListUserPermissions)" | Out-File UserAccessReport.csv -Append

            }

          }

          #Its a SharePoint Group, So search inside the group and check if the user is member of that group

         else

          {

                             foreach($user in $ListRoleAssignment.member.users)

                                 {

              if($user.LoginName -eq $SearchUser)

               {

                #Write-Host  "$SearchUser is Member of " $ListRoleAssignment.Member.Name "Group"

                 #Get the Group's Permissions on site

              $ListGroupPermissions=@()

                 foreach ($RoleDefinition  in $ListRoleAssignment.RoleDefinitionBindings)

                 {

                                $ListGroupPermissions += $RoleDefinition.Name +";"

                                }

              #write-host "Group has these permissions: " $ListGroupPermissions

              #Send the Data to Log file

              "$($Web.Url) `t Site `t $($List.Title)`t Member of $($ListRoleAssignment.Member.Name) Group `t $($ListGroupPermissions)" | Out-File UserAccessReport.csv -Append

             }

            }

         }

                       }

                }

              }

    }

   }

  }

#Call the function to Check User Access

GetUserAccessReport "http://sharepoint.crescent.com" "Global\Salaudeen"

Read more here