Microsoft SharePoint

SharePoint is a web-based collaborative platform that integrates natively with Microsoft 365. SharePoint is primarily sold as a document management and storage system, although it is also used for sharing information through an intranet, implementing internal applications, and for implementing business processes.

SharePoint Online Management Shell
SharePoint REST APIs

SharePoint Online Management Shell

The SharePoint Online Management Shell is a Windows PowerShell module that you can use to manage SharePoint settings at the organization level and site collection level.

To use SharePoint Online PowerShell commands, you must have the SharePoint Admin role or Global Administrator role in Microsoft 365.

Installation

Open PowerShell with administrative privileges.

powershell
Install-Module -Name Microsoft.Online.SharePoint.PowerShell

For the current user when no admin rights

Install-Module -Name Microsoft.Online.SharePoint.PowerShell -Scope CurrentUser

Update

Update-Module -Name Microsoft.Online.SharePoint.PowerShell
Get-Module -Name Microsoft.Online.SharePoint.PowerShell -ListAvailable | Select Name,Version

Connect

You need the “-admin” in the URL.

Connect-SPOService -Url https://<YOUR-COMPANY>-admin.sharepoint.com
Connect-SPOService -Url https://<YOUR-COMPANY>-admin.sharepoint.com -Credential my.user@<YOUR-COMPANY>.com

Older authentication method.

Connect-SPOService -Credential $creds -Url https://<YOUR-COMPANY>-admin.sharepoint.com -ModernAuth $true -AuthenticationUrl https://login.microsoftonline.com/organizations

List all sites

Open PowerShell with administrative privileges.

Import-Module Microsoft.Online.SharePoint.PowerShell

Run the following command to get all sites:

Get-SPOSite

SharePoint REST APIs

You can enter the “contentclass” directly in the SharePoint site search bar. E.g. “contentclass:STS_Site OR contentclass:STS_Web”

Values for contentclass:

  • STS_ListItem: List items in SharePoint lists. This includes individual items within document libraries, custom lists, calendar events, tasks, announcements, etc.
  • STS_Web: Sub-sites or child sites within a SharePoint site collection. These are individual sites that exist within a site hierarchy.
  • STS_List: SharePoint lists, including document libraries, custom lists, calendars, discussion boards, surveys, etc.
  • STS_Document: Individual documents within document libraries. This is often used in conjunction with STS_ListItem for finer-grained classification of content.
  • STS_Page: Wiki pages or web part pages within SharePoint sites.
  • STS_SiteAdminWebTemplate: Administrative sites or central administration sites within a SharePoint farm. These sites are used for managing and configuring SharePoint at a higher level.
  • STS_User: User profiles or user-related information within SharePoint. This can include user profiles, user properties, and other user-related data.
  • STS_Group: SharePoint groups or security groups within SharePoint. These are used for managing permissions and access control within SharePoint sites.

List all sites and privacy setting

Result might be in XML. For JSON, add HTTP header “Accept: application/json” to the HTTP request.

https://<YOUR-COMPANY>.sharepoint.com/_api/search/query?querytext='contentclass:STS_Site'&selectproperties='Title,Path,SitePrivacy'&rowlimit=1000

sharepoint-sites-enum.py

List all SharePoint sites and their privacy setting (when explicitly set). Also makes an HTTP request to validate.

#!/usr/bin/python
import requests
from requests_ntlm import HttpNtlmAuth
import getpass
import datetime
from time import sleep

# Remove warnings: InsecureRequestWarning: Unverified HTTPS request
requests.packages.urllib3.disable_warnings(requests.packages.urllib3.exceptions.InsecureRequestWarning)

# Construct the URL to get all sites and subsites
# STS_Site is for SharePoint sites, STS_Web is for subsites
api_url = "https://<REPLACE ME!>.sharepoint.com/_api/search/query?querytext='contentclass:STS_Site OR contentclass:STS_Web'&selectproperties='Title,Path,SitePrivacy'&rowlimit=1&startrow="

# Required cookies: rtFa and FedAuth
rtFa = "REPLACE ME!!!"
FedAuth = "REPLACE ME!!!"

headers = {
'Cookie' : 'rtFa=' + rtFa + '; FedAuth=' + FedAuth,
'Accept': 'application/json'
}

proxies = None
#proxies = {'http': 'http://127.0.0.1:8080', 'https': 'http://127.0.0.1:8080'}

# Ask for user input, for NTLM auth
print("Credentials are needed for NTLM authentication")
email = input("Enter your email: ")
password = getpass.getpass("Enter your password: ")

def send_request(url, delay=0, retry=2, message=""):
  if retry > 0:
    if delay > 0:
      print(message)
      sleep(delay)

    try:
      response = requests.get(url, verify=False, headers=headers, proxies=proxies, timeout=15)

      if response.status_code == 401: # Authenticate using NTLM if the server responded with "HTTP 401 Unauthorized"
        print("Info: Sending request using NTLM...")
        response = requests.get(url, verify=False, headers=headers, timeout=15, auth=HttpNtlmAuth(email, password))

      elif response.status_code == 503:
        # https://learn.microsoft.com/en-us/sharepoint/dev/general-development/how-to-avoid-getting-throttled-or-blocked-in-sharepoint-online
        response = send_request(url, 60, retry - 1, "Info: SharePoint throttling, waiting one minute...")

      elif response.status_code == 500:
        response = send_request(url, 5, retry - 1, "Info: Internal server error, retry sending request...")

      return response
    
    except requests.exceptions.Timeout:
      if retry > 0:
        return send_request(url, 30, retry - 1, "Info: timeout in send_request(" + url + ")")
      else:
        return None
    except requests.RequestException as e:
      print("Error: exception in send_request(" + url + ")", e)
      return None
  else:
    print("Info: Max retry reached for " + url)
    return None

i = 0
completed = False
print('Start building SharePoint site list: ' + str(datetime.datetime.now()))

with open('sharepoint-sites-subsites.csv', 'w', encoding="utf-8") as file:
  print("URL, Title, Server response")
  file.write("URL; Title; Server response\n")

  while(not completed):
    response = send_request(api_url + str(i))

    if response is not None and response.status_code == 200:
      data = response.json()
      row_count = data['PrimaryQueryResult']['RelevantResults']['RowCount']    
      sites = data['PrimaryQueryResult']['RelevantResults']['Table']['Rows']
    
      for site in sites:
        title = next(item['Value'] for item in site['Cells'] if item['Key'] == 'Title')
        url = next(item['Value'] for item in site['Cells'] if item['Key'] == 'Path')
        site_response = send_request(url)

        if site_response is not None:
          print(f"{i}: {url}; {title}; {str(site_response.status_code) + ' ' + site_response.reason}")
          file.write(f"{url}; {title}; {str(site_response.status_code) + ' ' + site_response.reason}\n")
        else:
          print(f"{i}: {url}; {title}; ERROR")
          file.write(f"{url}; {title}; ERROR\n")

      if row_count < 1:
        completed = True

    else:
        if response is not None:
          print("Error: status code " + str(response.status_code))
        
        completed = True

    i+=1

print('End building SharePoint site list: ' + str(datetime.datetime.now()))