Network Scanning

This is a side branch of my devlog series about my journey into MMO RPG network development (it all starts in mmodevlog.0). This time I got pulled underneath UDP/TCP sockets, into the low-level side of networking, and it turned out to be a useful detour: the better you understand what's going on down there, the more confident you feel writing the netcode itself. The task was simple - write a small program that scans the local network for available devices. While building it I learned a lot of new things, and that's what I want to share here.

Preparation

What do I actually want from the program? Honestly, a lot, but for the first version I'd just like it to print my local network address and list all available devices, ideally with a name telling me what kind of device each one is. To do that we need to handle a few things:

• Find the local network address and its mask
• Build a list of possible IP addresses in the network
• Check whether each IP address is reachable
• Figure out what device is hidden behind each IP address

Originally I wanted to make the app cross-platform (Unix and Windows), but in the end I decided to focus on Linux - low-level network work is heavily tied to the specific OS, and cross-platform support can be added later. As for the language, I of course went with Golang. And let me note upfront: everything below deals with IPv4 - it's simpler, and it's still the primary protocol on home networks.

The local network address and its mask

To get this information by hand, it's enough to know two commands - ipconfig on Windows and ifconfig on Unix systems. Then, from the interfaces printed to the console (and if you're a developer, you probably have quite a few), we can eyeball which one is our real internet-facing interface. We can spot it by the Default Gateway field: if it's present, it's most likely our local network address that faces outward. But how do we do this in a program?

The UDP-connect trick

Sure, we could parse the output of terminal commands, but that's uncomfortable and unreliable. So let me tell you about the first neat trick - determining the address via a UDP connection. But doesn't UDP work without a connection? Right, but in code we can emulate it: to "connect", the OS hands us basic info about the socket, and along with it, helps us determine our local address. Here's how it's done in Golang:

func retrieveLocalAddrIp() net.IP {
	addr, err := net.ResolveUDPAddr("udp4", "8.8.8.8:53")
	if err != nil {
		log.Fatal("Unable to resolve udp addr of 8.8.8.8:53", err)
	}

	udpConn, err := net.DialUDP("udp4", nil, addr)
	if err != nil {
		log.Fatal("Unable to resolve connect", err)
	}
	defer udpConn.Close()

	localAddr := udpConn.LocalAddr().(*net.UDPAddr)
	return localAddr.IP
}

We take any external routable address (I used Google's DNS, 8.8.8.8) and "connect" to it. The address being external isn't accidental: based on it, the OS picks the interface with the default route (the one facing the internet) and hands us its local address - all we do is grab it from the connection, and there's our local IP. Then we just walk through the network interfaces and find the one we need by comparing against this address. The most interesting part: the UDP "connection" does no handshake and doesn't send a single packet outward - meaning we determined our address without sending anything into the outside network. Example code:

type NetInterface struct {
	Interface net.Interface
	IP        net.IPNet
}

func retrieveLocalIpInterface(localAddr net.IP) (*NetInterface, error) {
	ints, err := net.Interfaces()

	if err != nil {
		return nil, err
	}

	for _, i := range ints {
		addrs, err := i.Addrs()
		if err != nil {
			log.Printf("Unable to retrieve interface addrs: %v\n", err)
		}

		for _, addr := range addrs {
			ipAddr, ok := addr.(*net.IPNet)
			if !ok {
				continue
			}

			if localAddr.Equal(ipAddr.IP) {
				return &NetInterface{
					Interface: i,
					IP:        *ipAddr,
				}, nil
			}
		}
	}

	return nil, nil
}

Using the standard library's built-in functions we get the list of interfaces, walk over each address on an interface, and if we find our local address (the one we got in the previous step) we return that interface along with extra info about the IP (like the mask).

And that's it! We've got the first piece of information we needed: the local IP address, the subnet mask, and our network interface.

Building the list of possible IP addresses

This part is simple - we just recall what a subnet mask even is. A subnet mask is a numeric value that defines the size of the network. To get the size of the network, say for a /24 mask, we need to remember that an IP is 4 bytes, or 32 bits. If the subnet mask is /24, that means 24 bits are allocated to the network address, and the remaining 8 bits can be taken by IP addresses inside that network. 8 bits is 256 addresses, where the first address is the network address and the last is the broadcast address; every address in between can potentially be occupied by a device on the network. Example code:

maskOnes, maskBits := addrWithMask.Mask.Size()
networkSize := uint32(1 << (maskBits - maskOnes)) // replaces math.Pow(2, maskBits-maskOnes) so we don't deal with floats

// start at 1 and stop at -1 to skip the network and broadcast addresses
for i := uint32(1); i < networkSize-1; i++ {
  ipUint32 += 1
  bytes := make([]byte, 4)
  binary.Encode(bytes, binary.BigEndian, ipUint32)
  ipByte := net.IP(bytes)

  // do something with the IP
}

By turning the IP into a uint32, we can run a plain loop that adds +1 and converts the number back into IP bytes, which lets us do whatever we want with this list of IPs.

Checking IP availability

Why a regular ping doesn't work

We have a list of possible IPs, and now for the easy part - just ping each device. We walk through the list and open a TCP connection to each IP with some timeout; if it doesn't answer within the allotted time, the device isn't on the network. You can try this and you'll see that, most likely, you'll end up with 0 devices on your network - except maybe the router. The reason is that the built-in firewalls on devices (your phone, your laptop) won't let you establish a connection to them over the local address; the firewall silently drops your packets, and you can't tell whether the device exists and just isn't answering, or whether it's not on the network at all.

ARP to the rescue

But there is still a way to reliably scan the network - though it only works within a single local segment (a single broadcast domain). The thing is, there's the OSI model with its 7 layers; for this article the 4-layer TCP/IP model is enough. And at layer 2, every packet is required to contain the MAC address of the device the packet is headed to. But while we can in theory know the IP ourselves, we almost never know the MAC address and have to get it from somewhere. And that's where ARP comes to the rescue.

ARP is a protocol used to determine the MAC address of another machine by its known IP address. And if a device wants to fully participate in the network, it must respond to an ARP request for its IP - otherwise other devices simply won't be able to reach it. Only the device whose IP is being requested responds; the rest ignore the request. Host firewalls usually filter traffic at layers 3-4 (IP/TCP), while ARP works at layer 2, so such a firewall generally doesn't touch it. You can go right now and run arp -a on Windows and see a list of addresses with their physical addresses - you might even spot your phone or laptop in the table already.

The easier way: ask the OS

But how do we trigger that request? As it turns out, when making any UDP/TCP/HTTP request, under the hood the OS first goes to its ARP table and checks whether it already has the MAC address; if not, it makes an ARP request to obtain it. So we can just fire off a UDP packet to every address, then check the ARP table and read all the info from there. That's what I did at first, and this approach has its pros and cons. The downside is that the timing of when an entry lands in the table isn't very clear, so you end up adding pauses in the code to make sure the OS has actually recorded all the addresses. But there's an upside: this approach doesn't require any special administrator privileges, since the OS does everything for us.

Building the packet by hand

But I went further. I thought - what if we build and send the ARP packet ourselves? I'd never worked with sending layer-2 packets before and figured it'd be a fun challenge. And honestly there's nothing hard about it at all - we just need to know the structure of the packet we have to assemble, and how to then send those raw bytes at layer 2.

The Ethernet header

An Ethernet frame starts with a small 14-byte header:

You might ask: how are we even supposed to get the Target MAC Address if the whole point of our request is to obtain that MAC address? Here's the thing - we want to learn a device's MAC address by its IP, so we need to poll every machine on the network: "If your IP is such-and-such, tell me your MAC address." How do we send a packet to all devices at once? That's what broadcast is for: send a packet to it and it gets forwarded to every device on the network, and only the specific device we're looking for will answer. The broadcast MAC address is ff:ff:ff:ff:ff:ff. For Ether Type we specify 0x0806 - this is the type of data that will be inside the packet, and since we want to send an ARP request, we specify the ARP type (0x0806). Example code for assembling the header:

broadcastMacAddr := []byte{0xff, 0xff, 0xff, 0xff, 0xff, 0xff}

ethPacketBytes := make([]byte, 0)
ethPacketBytes, _ = binary.Append(ethPacketBytes, binary.BigEndian, broadcastMacAddr)       // Target MAC
ethPacketBytes, _ = binary.Append(ethPacketBytes, binary.BigEndian, i.HardwareAddr)         // Sender MAC
ethPacketBytes, _ = binary.Append(ethPacketBytes, binary.BigEndian, uint16(unix.ETH_P_ARP)) // Ether Type (ARP 0x0806)

The ARP packet

Next comes the ARP packet itself, which is a bit bigger - 28 bytes - and consists of:

Hardware Type is the type of link-layer (layer 2) address we're resolving. The point is that ARP is a generic protocol and can work not only over Ethernet, so the packet has to explicitly state what kind of hardware addresses we're dealing with. For Ethernet this is 0x1, and the same value is used for Wi-Fi, since it uses the same 48-bit MAC addresses. Protocol Type is, by analogy, the type of network-layer (layer 3) address we want to resolve to a MAC - that is, the one that goes into the Sender/Target Protocol Address fields. We're working with IPv4, so we specify 0x0800 - the same value as the EtherType for IPv4. The next two fields follow automatically from these: since Hardware Type is Ethernet, Hardware Length is 6 (the length of a MAC address), and since Protocol Type is IPv4, Protocol Length is 4 (the length of an IPv4 address). Next is Operation - there are two options, 1 or 2: 1 means requesting a MAC address, 2 means replying, so we need 1. The sender fields are, I think, clear - we just plug in our MAC and our IP. But the Target fields raise the question again: the IP is obvious, but where do we get the MAC address this time - do we put broadcast again? It's actually simpler: since we're requesting the MAC address, this field is ignored, so we just write zeros into it. Example code:

zeroDistMacAddr := make([]byte, 6)
arpPacketBytes := make([]byte, 0)

arpPacketBytes, _ = binary.Append(arpPacketBytes, binary.BigEndian, uint16(unix.ARPHRD_ETHER)) // Hardware Type (Ethernet)
arpPacketBytes, _ = binary.Append(arpPacketBytes, binary.BigEndian, uint16(unix.ETH_P_IP))     // Protocol Type (IPv4 0x0800)
arpPacketBytes, _ = binary.Append(arpPacketBytes, binary.BigEndian, uint8(6))                  // Hardware Length (for Ethernet is 6)
arpPacketBytes, _ = binary.Append(arpPacketBytes, binary.BigEndian, uint8(4))                  // Protocol Length (for IPv4 is 4)
arpPacketBytes, _ = binary.Append(arpPacketBytes, binary.BigEndian, uint16(1))                 // Operation (1 for request, 2 for reply)
arpPacketBytes, _ = binary.Append(arpPacketBytes, binary.BigEndian, i.HardwareAddr)            // Sender hardware address (MAC)
arpPacketBytes, _ = binary.Append(arpPacketBytes, binary.BigEndian, sourceIp.To4())            // Sender Protocol Address (IP)
arpPacketBytes, _ = binary.Append(arpPacketBytes, binary.BigEndian, zeroDistMacAddr)           // Target hardware address (MAC)
arpPacketBytes, _ = binary.Append(arpPacketBytes, binary.BigEndian, distIp.To4())              // Target Protocol Address (IP)

packet := append(ethPacketBytes, arpPacketBytes...)

And that's it - nothing hard, the main thing is to figure out how to assemble the packet. Now all that's left is to send it and receive a response.

Sending the ARP packet

Since we want to send the packet at layer 2, we need to open a socket in the operating system that's responsible for sending raw packets. On Linux this is done as follows (AF_PACKET is a Linux-specific mechanism; on macOS/BSD raw layer-2 packets go through BPF, and on Windows it's different altogether):

fd, err := unix.Socket(unix.AF_PACKET, unix.SOCK_RAW, int(htons(unix.ETH_P_ARP))) // ARP Protocol ID 0x806
if err != nil {
  return err
}
defer unix.Close(fd)

Sending the packet and reading the response

The htons function is needed to convert the number to big-endian. Having opened the socket, we get a file descriptor, and using it together with the Sendto and Recvfrom functions we can send and receive data. It works like this:

sockAddr := unix.SockaddrLinklayer{
  Ifindex: i.Index,
}

if err := unix.Sendto(fd, packet, 0, &sockAddr); err != nil {
  return err
}

timeoutDuration := time.Millisecond * 100
tv := unix.Timeval{Sec: 0, Usec: timeoutDuration.Microseconds()}
unix.SetsockoptTimeval(fd, unix.SOL_SOCKET, unix.SO_RCVTIMEO, &tv)

recvPacket := make([]byte, 128)
readBytes, _, err := unix.Recvfrom(fd, recvPacket, 0)
if err != nil {
  if errors.Is(err, unix.EAGAIN) || errors.Is(err, unix.EWOULDBLOCK) {
    return ErrARPTimeout
  }

  return err
}

data := recvPacket[:readBytes]

Extracting the MAC address from the response

First we fill in the socket address struct - there are actually more fields, but for our purposes we don't need to fill them, since they either get default values or are taken directly from the bytes of the packet we assembled earlier. Before reading the response we set a timeout on the socket so we don't wait forever (otherwise nonexistent IPs simply won't answer and we'd wait indefinitely). If we got a response, we need to extract the MAC address. Recall the structure of the packets and think about where the MAC address is - it's in the ARP packet at the Sender hardware address field, so we read those 6 bytes at the right offset and then convert the bytes to hex, for a more familiar MAC address format (it would of course be better to add a separator too, but we'll skip that for now).

offsetToMacAddr := 14 + 2 + 2 + 1 + 1 + 2
log.Println(hex.EncodeToString(data[offsetToMacAddr : offsetToMacAddr+6]))

And there we have the MAC addresses of our devices. Let me note right away: we'll skip optimizations in this article. Right now we send one packet and block waiting for the response - which is slow. It would be better to open a single socket, fire all the packets through it at once, and set one shared timeout of, say, 500ms: whoever managed to answer in that time - we remember their MAC; whoever didn't - we consider the device absent.

Figuring out the device hidden behind an IP address

We've already done a huge amount of work - we scanned the network, found all the devices on it, and their MAC addresses too. But what if we go further and figure out what kind of device is hidden behind an IP address? It's actually not that hard: the first 3 bytes of a MAC address are a unique manufacturer identifier, so once we have the MAC address we can easily get the manufacturer using a vendor database - https://standards-oui.ieee.org/oui/oui.txt. Having preloaded the whole database into our code, this simple line lets us print the manufacturer:

log.Printf("MAC: %s\n", organizationNameByMac[strings.ToUpper(hex.EncodeToString(macAddr[:3]))])

Why not all devices get identified

Running this code on your own network, you'll probably notice that not all devices get identified - and this actually isn't a bug. I lied a little when I said identifying a device is very easy. In reality, many devices deliberately generate a MAC address that doesn't map to any vendor, for the sake of anonymity. For example, my iPhone generates a special MAC address that can't be used to tell what device it is. And I'll go further: the second-to-last bit of the first byte of the MAC address carries this information. If it's set to 0, the MAC address was assigned by IEEE and can be found in the database; if it's 1, the MAC address was generated by the device itself and we won't be able to tell what device it is. So the MAC-printing code turns into this:

if macAddr[0]&0x02 != 0 {
  log.Printf("MAC: %s\n", hex.EncodeToString(macAddr))
} else {
  log.Printf("MAC: %s\n", organizationNameByMac[strings.ToUpper(hex.EncodeToString(macAddr[:3]))])
}

Conclusion

And so we've finished the program. It came out small, but very interesting in terms of the knowledge I had to acquire or try out while building it. I'll say right away that there's a lot that could be improved: as I mentioned, optimizing the ARP request across all IP addresses, and making it cross-platform - since when you work with low-level things you have to deal with the OS a lot, and it's in our interest to extract common functions to abstract away the OS specifics. You could go further and finish the story of resolving names using DNS/mDNS, but these are all possible topics for future articles. I enjoyed figuring all of this out, learning and trying something new - and I hope you enjoyed learning it along with me.