Web Application Enumeration

WSTG-INFO: Information Gathering (OWASP Testing Guide)

Port scan

WSTG-INFO-04: Enumerate Applications on Webserver (OWASP Testing Guide)

When the pentest scope is a set of IP addresses, enumerate all web applications accessible through this target. Other services like FTP could also leave the web server vulnerable.

nslookup $DOMAIN

Port scan with Nmap (TCP, top 1000)

nmap -T4 -sT --top-ports 1000 $IP -oA nmap_tcp_top1000-$(date '+%Y.%m.%d.%Hh%M')
Quick Netcat port scan (from Nmap top 100) 
nc -nv -w 1 -z $IP 7 9 13 21-23 25-26 37 53 79-81 88 106 110-111 113 119 135 139 143-144 179 199 389 427 443-445 465 513-515 543-544 548 554 587 631 646 873 990 993 995 1025-1029 1110 1433 1720 1723 1755 1900 2000-2001 2049 2121 2717 3000 3128 3306 3389 3986 4899 5000 5009 5051 5060 5101 5190 5357 5432 5631 5666 5800 5900 6000-6001 6646 7070 8000 8008-8009 8080-8081 8443 8888 9100 9999-10000 32768 49152-49157

Infrastructure scan

Use tools like Nessus (free version allows scan of 16 IPs), OpenVAS, Tenable Security Center ($), Tsunami.

Spider/crawl

When a directory is discovered, relaunch a scan with this new directory path.

Also run on subdomains found in WSTG-INFO-01 with GHDB.

Tools: nmap, Burp Suite, Gobuster, Dirb, Dirbuster

Nmap

WSTG-CONF-04: Review Old Backup and Unreferenced Files for Sensitive Information (OWASP Testing Guide)

Discover backup files. See Backup and Unreferenced Files – Sensitive Information (WSTG-CONF-04).

Gobuster

Remember to look for metadata files. Use wordlist metadata.txt.

See Gobuster‘s Recommended scans.

Burp Suite

For APIs, use the Intruder module with Sniper or Cluster bomb payloads:

Use the Discover Content option in Burp Suite (Pro version only).

Review metadata for information leakage

WSTG-INFO-03: Review Webserver Metafiles for Information Leakage (OWASP Testing Guide)

Using Burp Suite, review any file previously found using the metadata.txt word list (robots.txt, humans.txt, security.txt, sitemap.xml, etc.).

Review metadata

Review “well-known” files

See Well-known URI (Wikipedia) for a description of services – well-known files are included in metadata.txt word list.

Review HTML tags

Review tags starting with "NO": "NOINDEX" and "NOFOLLOW"
Search for "<META NAME="ROBOTS"

Files describing APIs

See Password Attacks for generating wordlists based on regular expressions.

exrex "(/v[12])?(/(api|swagger|openapi))?/(api-docs(\.json)?|swagger(-ui(-openapi)?)?\.html|openapi\.html)"
exrex "(/v[12])?(/(api|swagger(-resources)?|openapi))?/(api-?docs(\.json)?|swagger(-ui(-openapi)?)?\.html|openapi\.html|swagger\.(json|yml))"
exrex "(/(api|swagger(-resources)?|openapi))?(/v[12])?/(api-?docs(\.json)?|swagger(-ui(-openapi)?)?\.html|openapi\.html|swagger\.(json|yml))"

Swagger

Search APIs on SwaggerHub. 

https://someurl:port/swagger-ui.html
https://someurl:port/management/swagger-ui.html

If swagger.json is found, use Swagger UI (GitHub) on Kali Linux with Apache:

cd /var/www/html
sudo wget -O swagger.json https://{DOMAIN}/swagger/v1/swagger.json
sudo nano swagger.json

Add the servers parameter with the URL of the server hosting the APIs (calls will be sent to this domain):

{
  "openapi": "3.0.1",
  "servers": [{ "url": "https://API-DOMAIN" }],
cd /var/www/html
sudo nano swagger.html
<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="utf-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1" />
  <meta
    name="description"
    content="SwaggerUI"
  />
  <title>SwaggerUI</title>
  <link rel="stylesheet" href="https://unpkg.com/swagger-ui-dist@4.5.0/swagger-ui.css" />
</head>
<body>
<div id="swagger-ui"></div>
<script src="https://unpkg.com/swagger-ui-dist@4.5.0/swagger-ui-bundle.js"></script>
<script>
  window.onload = () => {
    window.ui = SwaggerUIBundle({
      spec: location.host,
      url: 'swagger.json',
      dom_id: '#swagger-ui',
    });
  };
</script>
</body>
</html>

Access the API documentation at http://127.0.0.1/swagger.html.

WSDL (Web Service Description Language)

/path/to/copy/of/GoogleSearch.wsdl
http://webservice.example:1234/foo?WSDL
http://webservice.example:1234/index?WSDL

OpenID configuration

You can write OpenAPI definitions in YAML or JSON. Use an online converter to switch between formats if needed.

The members of this JSON object, called OpenID Connect provider metadata. OpenID Connect providers publish their metadata at a Well-Known URI which looks like this:

https://[base-server-url]/.well-known/openid-configuration

Example: https://accounts.google.com/.well-known/openid-configuration

Map the Application & Identify Entry Points & Features

WSTG-INFO-06: Identify Application Entry Points (OWASP Testing Guide)
WSTG-INFO-07: Map Execution Paths Through Application (OWASP Testing Guide)
WSTG-CONF-02: Test Application Platform Configuration (OWASP Testing Guide)
WSTG-CONF-05: Enumerate Infrastructure and Application Admin Interfaces (OWASP Testing Guide)

Send any interesting request to the Organizer module in Burp Suite and add a comment (e.g. “Scan request”, “Scan param x”, “Hidden param y”).

Map the web application

Take note of all the features found (admin portal, file upload/download, etc.).

  • Navigate the web application manually using Burp Suite and click on all buttons. Submit all forms.
  • Map the target application and understand the principal workflows.
  • Use results from the spider/crawl.
  • Identify all entry points.
  • If code source is available, inspect it to find other paths. Endpoint discovery in JS files (use regex), HTML comments (comments disclosing info about endpoints)
  • Manually identify possible entry and injection points through request and response analysis.
  • Identify where GETs are used and where POSTs are used.
Hidden parameters

Identify where GETs are used and where POSTs are used. Within the POST request, pay special attention to any hidden parameters sent by forms. Also keep note of parameters in GET requests as entry points.

  • Send a request to the Intruder module in Burp Suite.
  • Add a random parameter, like “?whatever=1”.
  • Make the variable name (“whatever”) the varying part.
  • Select the payload type Simple list.
  • Click Add from list and select Burp’s wordlist Server-side variable names.
  • Check for unusual behavior.
Test for debug parameters

Inspect parameters and try to enable debugging. Validate that no debugging code or extensions are left in the production environments.

Cookies

Identify where new cookies are set (Set-Cookie header), modified, or added to. Search cookies on Cookiepedia.

Review defaults and known files

Ensure that defaults and known files have been removed.

Search in developer comments

HTML comments are often used by the developers to include debugging information about the application. Sometimes they forget about the comments and they leave them on in production.

  • Browse the web app using Burp Suite.
  • In the Target tab, right-click on a target. This will NOT work in the Proxy tab.
  • Click on Engagement Tools->Find comments.
  • Review comments for interesting information.

Export comments to text file for easier search.

Testers should look for HTML comments which start with “”.
Check for SQL code, usernames and passwords, internal IP addresses, or debugging information. 

<!--Query: SELECT id, name FROM app.users WHERE active='1' -->
<!--Use the DB administrator password for testing: f@keP@a$$w0rD -->
<META name="Author" content="John Smith" -->
user
password
developpement
production

Look for display:none in html code to see if some features are in comments…

Admin interfaces

Identify hidden administrator interfaces and functionality.

  • Look at results from Gobuster and Burp Suite‘s scanner.
  • Manually try “/admin” or “/console”.
  • Each web framework may have its own admin default pages or path. Use information from previous reconnaissance and enumeration to target specific URL for these frameworks or technologies. See list of URL from for WSTG-CONF-05.

Check in HTML code for hidden fields (admin)

<input type="hidden" name="admin" value="no">

Check in cookies for administrator rights.

Cookie: session_cookie; useradmin=0

Web vulnerability scan

Test web vulnerability scanners on the Gin & Juice Shop. Use credentials carlos/hunter2.

Tools: Nikto, Burp Suite (Pro version), Nmap scripts

Quick Nikto scan (optional)

See Nikto.

nikto -host $URL -C all
nikto -host $URL -ssl -C all
URL="https://ginandjuice.shop/login"
nikto -id "user:password" -host $URL -ssl -C all

Source code scanning

If you have the source code, scan it using SAST tools.

Burp Suite

See Burp Suite. Other resources:
Using Burp Scanner during manual testing (PortSwigger)
Troubleshooting performance issues in Burp Suite (PortSwigger)

Authenticated scan

Before scanning, add your BChecks (create your own, or see GitHub) so they will be included in the scan (included by default). You can also scan with BChecks only – see Burp Suite.

  • Click on the Target tab.
  • Right-click on the target, click Scan.
  • Under Scan details:
    • Scan type: Crawl and audit
  • Under Scan configuration:
    • Select Balanced (results within a few hours, took 1h on the on the Gin & Juice Shop with 10 threads).
  • Under Application login:
    • Option 1: Select Use login credentials (username & password).
      • Click New.
      • Enter any label name (not used by the scan) and the credentials to log in the web application.
      • Click OK.
    • Option 2: Select Use recorded login sequences (record using Burp’s Chrome extension)
      • In the Proxy -> Intercept tab, click on Open browser.
      • Click on the extension icon and pin the Burp Suite Container extension if not already pinned.
      • Click on the Burp Suite Container icon and then on the Navigation Recorder tab.
      • Click on Start recording.
      • Do the login steps.
      • Click on the icon again and Stop recording.
      • The steps are copied in memory, paste them in the recorded login sequences in Burp.
  • Under Resource pool, create a new resource pool:
    • Maximum concurrent requests: 10
    • Select Automatic throttling with 429 (Too many requests) and 503 (Service Unavailable).
  • Click OK.
Scan specific requests

Interesting requests should be in the Organizer tab from previous manual crawling.

Find interesting requests and scan them. Right-click on a request from the Proxy tab (or other modules) and select Do active scan. Burp Scanner will use its default configuration to audit only this request.

Scan defined insertion points

Find requests with interesting or critical parameters (username, IDs, URL in parameter, file names, etc.). When there is more than one parameter to test in a request, see Burp Suite.

  • Install the Scan manual insertion point extension.
  • Highlight any sequence of characters within a request (typically a parameter value), right-click and select Extensions > Scan manual insertion point.
Scan APIs
  • From the Dashboard tab, click on New scan->API scan.
  • In API definition, upload the OpenAPI definition.

Validate findings

Validate all findings and filter out false positives.

Fingerprint Webserver, Framework & Components

WSTG-INFO-02: Fingerprint Web Server (OWASP Testing Guide)
WSTG-INFO-08: Fingerprint Web Application Framework (OWASP Testing Guide)
WSTG-INFO-09: Fingerprint Web Application (OWASP Testing Guide)
WSTG-INFO-10: Map Application Architecture (OWASP Testing Guide)

From previous scan outputs (Nmap, Nikto, Burp Suite), make a list of web server and all component versions. You can also use the Wappalyzer Firefox add-on to detect technologies that are used on a website.

  • Web servers
  • Databases
  • Libraries: inspect files in Burp Suite (like .js files), Burp scan results, and use Wappalyzer
  • Web Frameworks
  • CMS
  • Load balancers
  • CDN
  • Architecture like SaaS
  • Static cloud storage
  • Third party services used
  • Security controls like firewalls, IDS, IPS, WAF

Understand the architecture of the application and the technologies in use.

Web Frameworks

There are several most common locations to look in in order to define the current framework:

  • HTTP headers. When found, use VulnDB (GitHub) to report.
  • Cookies (search cookies on Cookiepedia). Some specific cookies like CAKEPHP are specific to the framework used.
  • HTML source code: look in meta tags, e.g. <meta name=”generator” content=”WordPress 3.9.2″ />
  • Specific files and folders, like /wp-login for WordPress.
  • File Extensions
  • Error Message
  • robots.txt can disclose paths giving clues

Web Application Firewalls (WAF)

Tools: wafw00f 

Load Balancer

Tools: lbd

Fingerprint Web Server (other options)

Tools: HTTPrint

Whatweb (BEST OPTION)

See WhatWeb.

URL="https://www.megacorpone.com"
whatweb "$URL"

Grab banner

curl -i $IP

Extract text

curl $IP -s -L | html2text -width '99' | uniq

Netcraft

Use the Netcraft Search DNS service to fingerprint the web server. Enter domain in search field on the right.

Netcat

Some sites might obfuscate the information in server response.

IP=x.x.x.x
PORT=80

# For HTTP 1.0: add Ctrl+v Ctrl+m => ^M
nc $IP $PORT
HEAD / HTTP/1.0^M

# For HTTP 1.1, always include Host and add Ctrl+v Ctrl+m => ^M
nc $IP $PORT
HEAD / HTTP/1.1^M
Host: hostname.domain.com^M
^M

# Response
HTTP/1.1 200 OK
Date: Mon, 16 Jun 2003 02:53:29 GMT
Server: Apache/1.3.3 (Unix) (Red Hat/Linux)
Last-Modified: Wed, 07 Oct 1998 11:18:14 GMT
ETag: "1813-49b-361b4df6"
Accept-Ranges: bytes
Content-Length: 1179
Connection: close
Content-Type: text/html

CMS Scan

Specific scan and configuration review depending on CMS: WordPress, Joomla!, Drupal

  • Test for outdated plugins
  • Test for XMLRPC exposure (WordPress)
  • Test for exposed admin portal

CMSmap

See CMSmap. Use authenticated scan when possible.

Specific Spider

Redo the spider using the CMS-specific wordlist.

Example 
wget https://raw.githubusercontent.com/danielmiessler/SecLists/master/Discovery/Web-Content/CMS/Drupal.txt
URL="https://example.com"
WL=Drupal.txt
gobuster dir -k -u $URL -w $WL -s '200,204,301,302,307,403,500' -b "" -e -d -z

Search for known vulnerabilities

Vulnerability description for reporting available in VulnDB (GitHub)

Previous vulnerability scans should have identified some vulnerabilities on identified components.

Review Webpage Content for Information Leakage

WSTG-INFO-05: Review Webpage Content for Information Leakage (OWASP Testing Guide)

These are also part of other tests. Some cleanup is required…

Common sources of information disclosure

Information disclosure can occur in a wide variety of contexts within a website. The following are some common examples of places where you can look to see if sensitive information is exposed.

  • Files for web crawlers
  • Directory listings
  • Developer comments (WSTG-INFO-05)
  • Error messages (WSTG-ERRH-01)
  • Debugging data
  • User account pages
  • Backup files (WSTG-CONF-04)
  • Insecure configuration (WSTG-CONF)
  • Version control history
META tags in HTML 
<META name="Author" content="John Smith">