Reconnaissance

Dr. Greg Bernstein

April 1st, 2021

Cyber Reconnaissance

Readings

Wojciech Mazurczyk and Luca Caviglione. 2021. Cyber reconnaissance techniques. Commun. ACM 64, 3 (March 2021), 86–95. DOI:https://doi.org/10.1145/3418293 main reference

Cyber Recon Taxonomy

Social Engineering: methods for collecting information to deceive a person or convincing him/her to behave in a desired manner.
Internet Intelligence: methods taking advantage of information publicly available in the Internet including databases accessible via the Web.
Network Information Gathering: methods for mapping the network (or computing) infrastructure of the victim.
Side-Channels: It groups methods exploiting unintended information leaked by the victim.

Cyber Recon Interaction/Detection

Open Source Intelligence (OSInt)

OSINT Definition 1

Derived from Wikipedia: OSINT

Open-source intelligence (OSINT) is the collection and analysis of information that is gathered from public, or open, sources. In the intelligence community, the term “open” refers to overt, publicly available sources (as opposed to covert or clandestine sources).

OSINT Definition 2

Derived from Wikipedia: OSINT

OSINT under one name or another has been around for hundreds of years. With the advent of instant communications and rapid information transfer, a great deal of actionable and predictive intelligence can now be obtained from public, unclassified sources. It is not related to open-source software.

OSINT Information Categories 1

Derived from Wikipedia: OSINT

Media, print newspapers, magazines, radio, and television from across and between countries.
Internet, online publications, blogs, discussion groups, citizen media, YouTube, and other social media websites.
Public Government Data, public government reports, budgets, hearings, telephone directories, press conferences, websites, and speeches.

OSINT Information Categories 2

Derived from Wikipedia: OSINT

Professional and Academic Publications, information acquired from journals, conferences, symposia, academic papers, dissertations, and theses.
Commercial Data, commercial imagery, financial and industrial assessments, and databases. Notes: this includes all types of fee based credit reporting and background check services.
Grey literature, technical reports, preprints, patents, working papers, business documents, unpublished works, and newsletters.

Internet Intelligence

Domains and IP addresses: www.arin.net, whois.icann.org
Internet scanners/databases: censys.io, www.shodan.io, www.zoomeye.org
General Repository of OSINT tools: OSINT Framework

censys.io

Censys started as an open source scanning and data processing tool called ZMAP, founded at the University of Michigan, with the mission of creating a public searchable database for the internet. Our team consists of industry veterans with expert Internet telemetry experience, delivering the most comprehensive internet scan data product and internet asset inventory in the market.

censys.io IPv4 Example

censys.io Certificate Example

ZoomEye

From ZoomEye About

ZoomEye is the leader of global cyberspace mapping, China’s first and world-renowned cyberspace search engine driven by 404 Laboratory of Knownsec, and also a world-famous cyberspace search engine. Through a large number of global surveying and mapping nodes, according to the global IPv4, IPv6 address and website domain name databases，it can continuously scan and identify multiple service port and protocols 24 hours a day, and finally map the whole or local cyberspace.

ZoomEye Example

All Web servers known to be running Express.js

What is Shodan?

From What is Shodan

Shodan is a search engine for Internet-connected devices. Web search engines, such as Google and Bing, are great for finding websites. But what if you’re interested in measuring which countries are becoming more connected? Or if you want to know which version of Microsoft IIS is the most popular? Or you want to find the control servers for malware? Maybe a new vulnerability came out and you want to see how many hosts it could affect? Traditional web search engines don’t let you answer those questions.

How does Shodan Work 1

From What is Shodan

Shodan gathers information about all devices directly connected to the Internet. If a device is directly hooked up to the Internet then Shodan queries it for various publicly-available information. The types of devices that are indexed can vary tremendously: ranging from small desktops up to nuclear power plants and everything in between.

How does Shodan Work 2

From What is Shodan

So what does Shodan index then? The bulk of the data is taken from banners, which are metadata about a software that’s running on a device. This can be information about the server software, what options the service supports, a welcome message or anything else that the client would like to know before interacting with the server.

Shodan Example

I tried a search for software that runs on small Linux devices…

Network Information Gathering

Network Information Gathering Categories 1

Traffic Sniffing: requires some type of network access
Scanning: Sends various types of packets to hosts to find open ports and services
Fingerprinting: Like scanning but tries to get more information on host (CPU, OS, software versions)

Network Information Gathering Categories 2

Enumeration: Similar to scanning but aimed at within private networks
Honeypot Detection: Additional techniques to detect “honeypots”

Traffic Sniffing

A passive technique, but requires some type of network access!
Very easy for an ISP or network owner, not so easy otherwise, i.e., in recon phase
This includes you on your machine, or your home network
Popular tools: TCPDump, WireShark

Example Port Mirroring

Network Scanning

nmap Nmap (“Network Mapper”) is a free and open source (license) utility for network discovery and security auditing.
nmap book
masscan This is an Internet-scale port scanner.

NMAP 1

Nmap (“Network Mapper”) is a free and open source utility for network exploration and security auditing. Many systems and network administrators also find it useful for tasks such as network inventory, managing service upgrade schedules, and monitoring host or service uptime. Nmap uses raw IP packets in novel ways to determine what hosts are available on the network, what services those hosts are offering, what operating systems they are running, what type of packet filters/firewalls are in use, and dozens of other characteristics. It was designed to rapidly scan large networks, but works fine against single hosts.

NMAP Scanning Phases 1

From NMAP Phases

Target Enumeration: “combination of host DNS names, IP addresses, CIDR network notations, and more”
Host discovery (ping scanning): “Network scans usually begin by discovering which targets on the network are online and thus worth deeper investigation”
Reverse-DNS resolution: “it looks up the reverse-DNS names of all hosts found online by the ping scan.” Can be skipped.

NMAP Scanning Phases 2

From NMAP Phases

Port scanning: “This is Nmap’s core operation. Probes are sent, and the responses (or non-responses) to those probes are used to classify remote ports into states such as open, closed, or filtered.”
Version detection: If any ports are found to be open, Nmap may be able to determine what server software is running on the remote system.
OS detection: If requested with the -O option, Nmap proceeds to OS detection

Host Discovery

From NMap Book: host-discovery

One of the very first steps in any network reconnaissance mission is to reduce a (sometimes huge) set of IP ranges into a list of active or interesting hosts. Scanning every port of every single IP address is slow and usually unnecessary. Network administrators may only be interested in hosts running a certain service, while security auditors may care about every single device with an IP address. An administrator may be comfortable using just an ICMP ping to locate hosts on his internal network, while an external penetration tester may use a diverse set of dozens of probes in an attempt to evade firewall restrictions.

NMAP Host Discovery Techniques

From NMap Book: host-discovery

There was a day when finding whether an IP address was registered to an active host was easy. Simply send an ICMP echo request (ping) packet and wait for a response. Firewalls rarely blocked these requests, and the vast majority of hosts obediently responded. Such a response has been required since 1989 by RFC 1122, which clearly states that “Every host MUST implement an ICMP Echo server function that receives Echo Requests and sends corresponding Echo Replies”.

Unfortunately for network explorers, many administrators have decided that security concerns trump RFC requirements and have blocked ICMP ping messages.

IP Packet Format

Internet Control Message Protocol (ICMP)

From Wikipedia: ICMP

The Internet Control Message Protocol (ICMP) is a supporting protocol in the Internet protocol suite. It is used by network devices, including routers, to send error messages and operational information indicating success or failure when communicating with another IP address, for example, an error is indicated when a requested service is not available or that a host or router could not be reached.

ICMP 2

General format of ICMP packet (carried in IP packet)

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|      TYPE     |      CODE     |             CHKSUM            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                            Content                            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
+

ICMP 3 – Famous ICMP Message Types

From Wikipedia: ICMP

Echo Reply (0), Echo Request (8)
Destination Unreachable (3) – 16 different codes including destination port unreachable
Router Advertisement (9), Router Solicitation (10)
Time Exceeded (11) – Including TTL expiration
Timestamp (13), Timestamp Reply (14)

NMAP Host Discovery Techniques 2

TCP SYN Ping (-PS): The -PS option sends an empty TCP packet with the SYN flag set. The default destination port is 80.
TCP ACK Ping (-PA): The TCP ACK ping is quite similar to the SYN ping. Such an ACK packet purports to be acknowledging data over an established TCP connection, but no such connection exists. So remote hosts should always respond with a RST packet, disclosing their existence in the process.

NMAP Host Discovery Techniques 3

The reason for offering both SYN and ACK ping probes is to maximize the chances of bypassing firewalls. Many administrators configure routers and other simple firewalls to block incoming SYN packets except for those destined for public services like the company web site or mail server. This prevents other incoming connections to the organization, while allowing users to make unobstructed outgoing connections to the Internet. This non-stateful approach takes up few resources on the firewall/router and is widely supported by hardware and software filters.

NMAP Host Discovery Techniques 4

UDP Ping (-PU): sends a UDP packet to the given ports. The port list takes the same format as with the previously discussed -PS and -PA options. If no ports are specified, the default is 40,125. A highly uncommon port is used by default because sending to open ports is often undesirable for this particular scan type. Upon hitting a closed port on the target machine, the UDP probe should elicit an ICMP port unreachable packet in return. This signifies to Nmap that the machine is up and available.

NMAP Host Discovery Techniques 5

ICMP Ping Types (-PE, -PP, and -PM): In addition to the unusual TCP and UDP host discovery types discussed previously, Nmap can send the standard packets sent by the ubiquitous ping program.
IP Protocol Ping (-PO): The newest host discovery option is the IP protocol ping, which sends IP packets with the specified protocol number set in their IP header. The protocol list takes the same format as do port lists in the previously discussed TCP and UDP host discovery options. If no protocols are specified, the default is to send multiple IP packets for ICMP (protocol 1), IGMP (protocol 2), and IP-in-IP (protocol 4).
ARP Scan (-PR): One of the most common Nmap usage scenarios is to scan an ethernet LAN. On most LANs, especially those using private address ranges granted by RFC 1918, the vast majority of IP addresses are unused at any given time. When Nmap tries to send a raw IP packet such as an ICMP echo request, the operating system must determine the destination hardware (ARP) address corresponding to the target IP so that it can address the ethernet frame properly.

Reminder UDP Header

The User Datagram Protocol (UDP) is carried in IP packets

Reminder TCP Header

The Transport Control Protocol (TCP) is carried in IP packets

TCP and UDP Ports 1

From NMAP book – ports

well-known ports

These are reserved ports (within the range of 1 to 1,023, as discussed above) which have been registered with the IANA for a certain service. Familiar examples are ports 22, 25, and 80 for the services SSH, SMTP, and HTTP, respectively.

registered ports

These ports fall within the range 1,024 to 49,151 and have been registered with the IANA in the same way the well known ports have. Most of these are not as commonly used as the well-known ports. The key difference is that unprivileged users can bind to these ports and thus run the services on their registered port. Users cannot do so on most platforms for well-known ports, since they reside in the reserved port range.

TCP and UDP Ports 2

From NMAP book – ports

dynamic and/or private ports

The IANA reserves the port numbers from 49152 through 65535 for dynamic uses such as those discussed in the ephemeral ports section. Proprietary services that are only used within a company may also use these ports.

Port Scanning 1

There are numerous practical benefits to regularly scanning your networks. Foremost among these is security. One of the central tenets of network security is that reducing the number and complexity of services offered reduces the opportunity for attackers to break in. Most remote network compromises come from exploiting a server application listening on a TCP or UDP port. In many cases, the exploited application is not even used by the targeted organization, but was enabled by default when the machine was set up. Had that service been disabled, or protected by a firewall, the attack would have been thwarted.

Some NMAP Scanning Techniques

A selection of NMAP port scanning techniques

TCP SYN Scan: Only sends the TCP SYN segment and does not complete the TCP handshake. Only port status is revealed.
TCP Connect Scan: Tries to make a full TCP connection.
UDP Scan: Scans UDP ports
TCP Idle Scan: See idle scan This is a very stealthy scan that involves an intermediate system to avoid revealing scanning machine to target.

Version and Service Detection 1

From Version and Service Detection

Once the TCP connection is made, Nmap listens for roughly five seconds. Many common services, including most FTP, SSH, SMTP, Telnet, POP3, and IMAP servers, identify themselves in an initial welcome banner. Nmap refers to this as the “NULL probe” because Nmap just listens for responses without sending any probe data.

Version and Service Detection 2

From Version and Service Detection

If any data is received, Nmap compares it to about 3,000 “NULL probe” signatures in its nmap-service-probes file (described in the section called “nmap-service-probes File Format”). If the service is fully identified, we are done with that port! The regular expression in signatures can includes substring matches to pick version numbers out of the response. In some cases, Nmap gets a “soft match” on the service type, but no version info. In that case, Nmap continues but only sends probes that are known to recognize the soft-matched service type.

Version and Service Detection 3

From Version and Service Detection

(Otherwise) Nmap sequentially (in the order they appear in the file) executes the probe(s) that match the port number being scanned. Each probe includes a probe string (which can be arbitrary ASCII text or escaped binary), which is sent to the port. Responses that come back are compared to a list of signature regular expressions of the same type as discussed in the NULL probe description above.

Version and Service Detection 4

From Version and Service Detection

As with the NULL probe, these tests can either result in a full match (ends processing for the remote service), a soft match (limits future probes to those which match a certain service), or no match at all. The exact list of regular expressions that Nmap uses to test for a match depends on the probe fallback configuration. For instance, the data returned from the X11Probe is very unlikely to match any regular expressions crafted for the GetRequest probe.

Fingerprinting 1

From Cyber recon techniques

Scanning can be also used to recognize the guest OS or the applications available in the target nodes. This is known as fingerprinting. For the case of OS fingerprinting, the main technique exploits the fact that the network stack of each OS exhibits minor differences when replying to well-crafted probe packets (for example, the initial sequence number of the TCP segments, the default TTL value for ICMP packets, among others). Such artifacts can be utilized to remotely determine the type and version of the OS of the inspected device.

Fingerprinting 2

From Cyber recon techniques

Application fingerprinting uses a slightly different technique. In this case, the attackers take advantage of a “banner”, which is a sort of preamble information that a server-side application sends before accepting a client. By stimulating a host with connection requests, they can harvest banners to reveal details on the active applications and services (for example, the version of the software can be used to determine the known vulnerabilities).

Some Reasons for OS Detection

From NMAP Book– OS Detection

Determining the Vulnerability of Target Hosts
Tailoring Exploits
Network Inventory and Support
Detecting Unauthorized and Dangerous Devices
Social Engineering – How?

OS Detection Techniques used by NMAP

From NMAP Book– OS Detection

Probes Sent: Sequence generation (SEQ, OPS, WIN, and T1), ICMP echo (IE), TCP explicit congestion notification (ECN), TCP (T2–T7), UDP (U1)
Response Tests: TCP ISN greatest common divisor (GCD), TCP ISN counter rate (ISR), TCP ISN sequence predictability index (SP), IP ID sequence generation algorithm (TI, CI, II), Shared IP ID sequence Boolean (SS), TCP timestamp option algorithm (TS), TCP options (O, O1–O6), TCP initial window size (W, W1–W6), Responsiveness (R), IP don’t fragment bit (DF), Don’t fragment (ICMP) (DFI), IP initial time-to-live (T), IP initial time-to-live guess (TG), Explicit congestion notification (CC), TCP miscellaneous quirks (Q), TCP sequence number (S), TCP acknowledgment number (A), TCP flags (F), TCP RST data checksum (RD), IP total length (IPL), Unused port unreachable field nonzero (UN), Returned probe IP total length value (RIPL), Returned probe IP ID value (RID), Integrity of returned probe IP checksum value (RIPCK), Integrity of returned probe UDP checksum (RUCK), Integrity of returned UDP data (RUD), ICMP response code (CD)

Nmap Firewall Detection/Evasion Techniques

From Detecting and Subverting Firewalls and Intrusion Detection Systems

Determining Firewall Rules: Standard SYN Scan, Sneaky firewalls that return RST, ACK Scan, IP ID Tricks, UDP Version Scanning
Bypassing Firewall Rules: Exotic Scan Flags, Source Port Manipulation, IPv6 Attacks, IP ID Idle Scanning, Multiple Ping Probes, Fragmentation, Proxies, MAC Address Spoofing, Source Routing, FTP Bounce Scan, Take an Alternative Path

Nmap IDS Detection/Evasion Techniques

From Detecting and Subverting Firewalls and Intrusion Detection Systems

Intrusion Detection System Detection: Reverse probes, Sudden firewall changes and suspicious packets, Naming conventions, Unexplained TTL jumps

Avoiding Intrusion Detection Systems: Slow down, Scatter probes across networks rather than scanning hosts consecutively, Fragment packets, Evade specific rules, Avoid easily detected Nmap features

Misleading Intrusion Detection Systems: Decoys, Port scan spoofing, Idle scan, DNS proxying

Home Network Inventory Example

Dynamic Host Configuration Protocol (DHCP)

From Wikipedia: DHCP

The Dynamic Host Configuration Protocol (DHCP) is a network management protocol used on Internet Protocol (IP) local area networks. A DHCP server must be present on the network. A device connected to the network requests an IP address from the DHCP server using the DHCP protocol; the server assigns a unique address to the device, identifying it for TCP/IP communication, and supplies other network configuration parameters.

Home Network DHCP Leases

From my home router March 2021, 11 devices have addresses

Nmap Host Detection

Nmap scan 192.168.1.1-255 March 2021, 9 devices found

What is it? 1

Nmap can’t identify the OS/device, can you?

What is it? 2

Nmap can’t identify the OS/device, can you?

IoT Detection

OS Detection: Linux

OS Detection: Windows

Vulnerability Scanning

Scanners and vulnerability databases

We’ve seen that scanners like NMap can discover machines on our networks and what services and operating systems are running on them.
We’ve seen vulnerability databases such as NVD and CVE
Vulnerability scanners combine the two concepts

Typical Commercial Tools

Nessus: “Nessus is the #1 deployed solution for application vulnerability assessment”
Acunetix: “Acunetix is an application security testing solution for securing your websites, web applications, and APIs”
Vulners “Vulnerability Assessment Platform”

Open Source Tools 1

OpenVAS: “OpenVAS is a full-featured vulnerability scanner. Its capabilities include unauthenticated testing, authenticated testing, various high level and low level Internet and industrial protocols, performance tuning for large-scale scans and a powerful internal programming language to implement any type of vulnerability test.”
Subgraph Vega: “Vega is a free and open source web security scanner and web security testing platform to test the security of web applications. Vega can help you find and validate SQL Injection, Cross-Site Scripting (XSS), inadvertently disclosed sensitive information, and other vulnerabilities. It is written in Java, GUI based, and runs on Linux, OS X, and Windows.”

Open Source Tools 2

OWASP Zed Attack Proxy (ZAP) “Zed Attack Proxy (ZAP) is a free, open-source penetration testing tool being maintained under the umbrella of the Open Web Application Security Project (OWASP). ZAP is designed specifically for testing web applications and is both flexible and extensible.”