1  Creating Networks

From the simplest two host network to complex enterprise or carrier networks spanning the globe a network still comes down to nodes (switches and hosts) connected by links. One of the most fundamental network design and deployment decisions comes down to what to build/own and what to rent/lease. Typically we don't get much choice on whether we get to own or rent links or nodes, but we can have a wide choice in what technologies to use.

Once we need our network to extend beyond a single building we generally have to use other peoples networks to create the links we need. For example carriers offer many different WAN services as this example network map the the carrier Centurylink shows.

1.1  Issues with using other peoples networks

1.1.1 Incompatibility of address types

    1.  Example: A company uses private IPv4 addresses but wants to interconnect sites across the public internet.   For more detail: RFC1918 specifies private IPv4 address ranges.

    2. Related Example: Multiple users within a private address space need to share a singe public IP address → NAT (Network address translation).  See Wikipedia and IETF RFC2663, section 4.1.2 "Network Address Port Translation (NAPT)" is what we usually just call NAT.

      “NAPT extends the notion of translation one step further by
      also translating transport identifier (e.g., TCP and UDP
      port numbers, ICMP query identifiers). This allows the
      transport identifiers of a number of private hosts to be
      multiplexed into the transport identifiers of a single
      external address. NAPT allows a set of hosts to share a
      single external address.”

1.1.2 Compatibility of signal types

Example: I want to support a legacy telephone system and the only network infrastructure available is IP based. A standardized technique for doing this over IP or MPLS is called psuedowires

  • Pseudowire architecture RFC3985

  • Structure-Aware Time Division Multiplexed (TDM) Circuit Emulation Service over Packet Switched Network CESoPSN. See section 5.2 as an illustration of a straight forward way of packing DS0 signals into a packet. Also use this to illustrate compromises between latency and efficiency.

1.1.3 Incompatibility of layers

Example: I want to extend an Ethernet LAN between two sites and the only network available between this is an IP network.

2 Adaptation, Layering, and Tunneling

As part of normal network operations we expect to add layers of extra functionality above the link and network layer protocols we use. In fact both Ethernet Frames and IP datagrams contain explicit areas in their headers to tell the reciever what type of payload they are carrying. See Figures 1 and 2. In addition the next layer protocol may need to do some extra work to aid in the transmission over the lower level. We generally call this process adaptation.

Figure 1: Ethernet frame, highlighting the type field.
Figure 1: Ethernet frame, highlighting the type field.

From RFC7042 we get a sampling of Ethertypes:

   0x0800  Internet Protocol Version 4 (IPv4)
   0x0806  Address Resolution Protocol (ARP)
   0x0808  Frame Relay ARP
   0x22F3  TRILL
   0x22F4  L2-IS-IS
   0x8035  Reverse Address Resolution Protocol (RARP)
   0x86DD  Internet Protocol Version 6 (IPv6)
   0x880B  Point-to-Point Protocol (PPP)
   0x880C  General Switch Management Protocol (GSMP)
   0x8847  MPLS
   0x8848  MPLS with upstream-assigned label
   0x8861  Multicast Channel Allocation Protocol (MCAP)
   0x8863  PPP over Ethernet (PPPoE) Discovery Stage
   0x8864  PPP over Ethernet (PPPoE) Session Stage
   0x893B  TRILL Fine Grained Labeling (FGL)
   0x8946  TRILL RBridge Channel
Figure 2: IPv4 header, highlighting the protocol field.
Figure 2: IPv4 header, highlighting the protocol field.

From IANA here are a sampling of IPv4 protocol numbers:

1   ICMP    Internet Control Message
2   IGMP    Internet Group Management
3   GGP     Gateway-to-Gateway
4   IPv4    IPv4 encapsulation
5   ST      Stream
6   TCP     Transmission Control
7   CBT     CBT
8   EGP     Exterior Gateway Protocol
9   IGP     any private interior gateway
10  BBN-RCC-MON     BBN RCC Monitoring
11  NVP-II  Network Voice Protocol
12  PUP     PUP
15  XNET    Cross Net Debugger
16  CHAOS   Chaos
17  UDP     User Datagram

2.2 More on Adaptation

With PPP we saw the type of effort needed to put packets over bit or byte streams, i.e., we needed to introduce some framing to delimit the start and end of the packet and such. What about if we want to carry old fashioned voice DS0 circuits over a packet network? Couldn't we just gather a bunch of bytes from the DS0s and put them into a packet? Actually that is kind of what is done (Figures 3 and 4).

Figure 3: Packet diagram for Circuit Emulation Service (CES).
Figure 3: Packet diagram for Circuit Emulation Service (CES).

One tricky part of putting "circuits" over "packets" is timing recovery, i.e., making sure that the bytes get played out at the right speed at the other end of the network. From the above diagram we see that the CES service uses the Real-time Transport Protocol to help with this function.

Figure 4: Packet of DS0 bytes into CES payload.
Figure 4: Packet of DS0 bytes into CES payload.

From looking at the the previous headers for CES one should wonder about the efficiency of such an approach. When we talk about the efficiency of a digital communication scheme we generally mean the ratio of payload bytes (or bits) transported to the total bits transported.

Example 1: Consider an old fashioned T1 frame that carries 24 DS0 bytes along with 1 bit of synchronization overhead. The efficiency of such a scheme is:

\[ Efficiency = \frac{24 \times 8}{24 \times 8 + 1} = 99.482\% \]

Example 2: Consider an Ethernet frame with 1200 bytes of payload. Assuming no padding we have 7 octets of preamble, 1 octet for SFD, 6 octets for destination address, 6 octets for source address, 2 octets for type, and 4 octets for the frame check sequence. This gives a total of 26 bytes of overhead. Hence the efficiency for this packet transmission is:

\[ Efficiency = \frac{1200 bytes}{1200 bytes + 26 bytes} = 97.88\% \]

What if the payload was only 100 bytes?

\[ Efficiency = \frac{100 bytes}{100 bytes + 26 bytes} = 79.37\% \]

For better efficiency we want to have large payloads for our packets. It looks like our old fashioned T1 line is more "efficient" than modern Ethernet transmission? Or is more analysis and context needed to make the assesment?

2.2.1 Latency in Real Time Communications

For real time communications we have an additional issue that can get in the way of efficiency and that is latency.

Example: For a 64 Kbps voice channel how long does it take to buffer up 100 bytes for transmission? What about 1000 bytes? Answer:

\[ time = \frac{100 bytes \times 8 bits/byte}{64Kbps} = 12.5msec \]

\[ time = \frac{1000 bytes \times 8 bits/byte}{64Kbps} = 125.0msec \]

For comparison the approximate distance from the US to India is 13600km and the speed of light is approximately 300,000km/s which would give a delay of approximately 45.3msec. Hence to achieve the highest efficiency in filling an Ethernet packet with DS0 voice samples would result in a delay equivalent to about three times the fiber optic transmission delay from the US to India. Hence we see that for real time communications there is a trade off between efficiency and latency.

2.3 What's Tunneling?

As we've seen with "normal" protocol layering, we frequently encapsulate payloads into additional "containers" to help "adapt" that payload to the link or network layer packet format. For example shown in Figures 5 and 6 are the UDP and TCP transport layer headers used with the IP network layer.

Figure 5: User Datagram Protocol (UDP) header.
Figure 5: User Datagram Protocol (UDP) header.
Figure 6: Transport Control Protocol (TCP) header.
Figure 6: Transport Control Protocol (TCP) header.

So how is tunneling different from the usual encapsulation and adaptation that we see in our layer protocol stacks? In general we use the term tunneling when the payload being encapsulated is a packet (or signal) from a layer equal to or lower than that the one we will be using to carry that packet or signal. We'll see or have seen examples of the following:

  • Physical layer — DS0s — over IP
  • Link Layer — Ethernet — over IP
  • IP — say with private address — over IP — with public addresses

Typical network architecture with tunneling providing interconnection:

Figure 7: Tunneling across the internet.
Figure 7: Tunneling across the internet.

Network Address Translation (NAT), Half a Tunnel?: Somewhat similar to tunneling is NAT, Figure 8, which from a network diagram looks like half a tunnel.

Figure 8: Half a tunnel with NAT.
Figure 8: Half a tunnel with NAT.

2.3.1 Middleboxes

In the tunnel network diagram, Figure 7, we added special boxes, labeled TE, to denote the tunnel end points. Such processing may take place in a separate piece of network equipment or may be combined with other network equipment such as a router. In either case the tunneling functionality is very different from routing. In a home router typically NAT, Firewall, IP routing, Ethernet Switching, and wireless access point are all combined in the same physical box.

Things like tunnel endpoints, NAT, and Firewalls may exist in the IP "path" between source and destination and have different functionality than IP routers. At the IETF such entities are called middleboxes. RFC3234 discusses why these have become so important to the architecture of the internet and describes approximately twenty different types of middleboxes.

4 Virtual Private Networks

A nice set of definitions for VPNs and the problem they solve comes from RFC3809 1, June 2004:

   Corporations and other organizations have become increasingly
   dependent on their networks for telecommunications and data
   communication.  The data communication networks were originally built
   as Local Area Networks (LAN).  Over time the possibility to
   interconnect the networks on different sites has become more and more
   important.  The connectivity for corporate networks has been supplied
   by service providers, mainly as Frame Relay (FR) or Asynchronous
   Transfer Mode (ATM) connections, and more recently as Ethernet and
   IP-based tunnels.  This type of network, interconnecting a number of
   sites over a shared network infrastructure is called Virtual Private
   Network (VPN).  If the sites belong to the same organization, the VPN
   is called an Intranet.  If the sites belong to different
   organizations that share a common interest, the VPN is called an

4.1 Network Contexts

The network context — who is connecting to whom, for what reason, etc... — greatly influences the types of problems encountered in creating a VPN and the technologies utilized.

One decomposition of the VPN networking problem from 2:

  • Remote Access VPN:
  • Site-to-Site VPN:
    • Intranet VPNs connecting sites within an organization
    • Extranet VPNs connecting two different organizations

Another decomposition could be based on who provisions the VPNs:

  • Provider Provisioned VPNs (PPVPN)
    • The bulk of recent standard activitions at the IETF and IEEE have be focused on PPVPNs.
  • Customer or Enterprise Provisioned VPNs
    • Based on user configured tunnels
    • Based on enterprise configured IEEE 802.1Q (Ethernet) VLANs

Still another decomposition can be based on the layer that the VPN provides service. At one point in time the IETF had three different working groups for layer 1, 2, and 3 VPNs respectively.

Finally when the VPN is offered by a service provider (carrier) they may offer different types of services such as:

  • Virtual Private Wire Service (VPWS) — like the point to point tunnels we've looked at.
  • Virtual Private LAN Service (VPLS) — which takes care of getting an Ethernet frame to the appropriate location as if all locations were on a single LAN.

4.2 Ethernet VLANs

Fundamental ideas of Ethernet VPNs:

  1. Allow multiple Virtual “LANs” (really bridged networks) to share the same physical infrastructure. With each VLAN given a unique Identifier, known as a VID (VLAN identifier)

  2. Restrict forwarding and broadcasting to individual VLANs, i.e., provides traffic isolation within a common network.

  3. Enhances learning bridges to understand VLAN tags and such.

  Examples: separate departments within a company, multiple tenant office spaces, multiple client applications within a data center.

4.2.1 VLAN Implementation Outline

The first thing we need to enable VLANs is a place in the Ethernet frame header to put the VLAN ID (VID). Recall the Ethernet frame structure shown in Figure 12.

Figure 12: Ethernet Frame
Figure 12: Ethernet Frame

It seems that there isn't any space for a VID, i.e., all the fields are used. The trick that that IEEE adopted was to define new Ethertypes to support VLANs and then appending the "regular" Ethertype, i.e., the one that indicates the upper layer protocol as part of the payload as shown in Figure 13.

Figure 13: VLAN Ethernet from Wikipedia
Figure 13: VLAN Ethernet from Wikipedia

After the destination and source address a “tag header” is inserted and the FCS is recomputed. The header now consists of the Tag Protocol identifier (TPID) — essentially a new EtherType — and Tag Control Information (TCI).

Figure 14 shows a list of the VLAN EtherTypes (from IEEE 802.1Q-2011, this is 1365 page document):

Figure 14: VLAN tag control information field
Figure 14: VLAN tag control information field

In addition to the VID = VLAN Identifier the Tag Control Information (TCI) field contains congrestion and QoS related subfields: DEI = Drop Eligible Indicator and PCP=Priority Code Point as shown in Figure 15.

Figure 15: Drop, Priority, and VID
Figure 15: Drop, Priority, and VID

Formally 802.Q describes VLAN functionality via an additional layer called the “Enhanced Internal Sublayer Service”.

4.2.2 How does Frame "tagging" work?

Who inserts VLAN tags onto packets?

Switches do!  Switches can be configured to assign VIDs based on port or protocol.

How are VLANs configured in a typical case?

  1. Two fundamental VLAN switch port types: “tagged” and “untagged”.

  2. Some ports will be “access” ports

  3. Some ports will be “trunk” (switch-to-switch ports)

  4. Configure “access” ports as “untagged” and set the PVID (port VLAN ID) to the VID of the VLAN that the host should belong to.  Untagged frames from the host will get the PVID added when entering the bridged network. Tagged frames for the host will have their tags removed.

  5. Configure “trunk” ports as “tagged”.  Such ports can belong to multiple VLANs and the tags are needed to restrict forwarding.

  6. How can we get traffic to pass between VLANs?  (IP) Route it!

    1. Create a “tagged” port on one of the switches and configure it to belong to all VLANs that you want to route between.

    2. Hook up a “lollipop” (aka “one-armed”) router to this multi-VLAN port.

    3. Note that most Linux distributions have support for IEEE 802.Q tagging. In current Ubuntu this configured with the vconfig command that comes with the vlan package. Hence you could create an inexpensive lollipop router out of Linux.

Example low cost VLAN capable switch:



4.2.3 Bandwidth Efficiency via MSTP

Ethernet VLANs help us achieve traffic isolation and restrict broadcast domains but what about better link utilization? The problem with bridged networks from a bandwidth/link utilization perspective is that they are tree topology networks! Consider the network shown in Figure 16 where the circles denote hosts and the color represents the logical grouping of the hosts, i.e., we want to put the blue and orange colored hosts into separate VLANs.

Figure 16: Physical Ethernet topology, hosts color coded by grouping.
Figure 16: Physical Ethernet topology, hosts color coded by grouping.

Ethernet spanning tree protocol prunes links that would result in loops, i.e., they are not used, and could give us a logical connectivity for this network as shown in Figure 17. A key property of tree topologies is that there is only one path between any two nodes. Looking at Figure 17 we see that for some of the blue hosts to communicate with their peers will require paths traversing up to four links and this traffic will cross links used by the orange nodes to communicate.

Figure 17: Logical Ethernet single tree topology.
Figure 17: Logical Ethernet single tree topology.

The multiple spanning tree protocol (MSTP) feature in 802.1Q allows us to assign different spanning trees to different VLANs. In our example network we could come up with different trees for the orange and blue VLANs as shown in Figure 18 where the different link colors are used to denote the two different trees in use and the the VLANs using them.

Figure 18: Ethernet VLANs and multiple trees.
Figure 18: Ethernet VLANs and multiple trees.

The advantage of this VLAN to tree assignment is even more obvious when we prune off links in each VLAN tree that will not be carrying traffic for that VLAN as shown in 19 where we can see that each link in this network is carrying traffic for only one VLAN. As we will see a bit more when we study SDNs that general mesh networks possess a great many trees and that trees play an important role in different traffic forwarding paradigms. In addition, SDNs will give us the most freedom to choose forwarding trees.

Figure 19: Ethernet VLANs and traffic carrying tree links.
Figure 19: Ethernet VLANs and traffic carrying tree links.

4.3 Notes on IETF VPN Efforts

Not required reading

The bulk of the IETF work on VPNs has concluded. It had in the past four working groups related to VPNs:

Figure 20: L1VPN reference model
Figure 20: L1VPN reference model
Figure 21: L2VPN Virtual Private Wire Service (VPWS) reference model RFC4664.
Figure 21: L2VPN Virtual Private Wire Service (VPWS) reference model RFC4664.
Figure 22: L2VPN Virtual Private LAN service (VPLS) reference model RFC4664.
Figure 22: L2VPN Virtual Private LAN service (VPLS) reference model RFC4664.
Figure 23: L2VPN Virtual Private LAN service (VPLS) reference model with stack from RFC4664.
Figure 23: L2VPN Virtual Private LAN service (VPLS) reference model with stack from RFC4664.


[1] A. Nagarajan, “Generic Requirements for Provider Provisioned Virtual Private Networks (PPVPN),” Jun-2004. [Online]. Available: https://tools.ietf.org/html/rfc3809.

[2] N. F. Mir, Computer and communication networks, Second edition. Upper Saddle River, NJ: Prentice Hall, 2015.