Click here to Skip to main content
Click here to Skip to main content

Developing a SOHO HTTP filter step by step

, 26 Aug 2007 CPOL
Rate this:
Please Sign up or sign in to vote.
The main goal of this article is to explain the practical details of low level network security programming.

Screenshot - screenshot.gif

Screenshot - blocked.jpg

1. Abstract

Most web filters work inline, meaning that all outgoing and incoming packets are passed through a filter driver. While this approach has its benefits, it has a big flaw: the filtering process affects the data transfer throughput. This project presents an experimental remedy to this issue by putting the filter engine in sniffer mode. This way, the filtering process and the data transfer act independently.

2. Requirements

  • The article expects the reader to be familiar with C++ and TCP/IP concepts.
  • The source code uses the following libraries:
    • Winsock.
    • Ethereal: A packet capture and network analyzer.
  • The code was compiled and built with VC7 on Windows Server 2003.

3. Introduction

The main goal of this article is to explain the practical details of low level network programming. There are many commercial and open source firewalls and HTTP filters available on both Linux and Windows. But, internally, most of them follow the same approach to finding their targets. The References section of this article provides you handy books related to this topic.

Particularly, a web filter could act in two modes to inspect outgoing packets for blacklist keywords: Inline mode and Sniffer mode. We also explain both these modes of operation and compare them.

4. Background

This section explains the TCP/IP protocol stack, the HTTP protocol behavior, and the Boyer-Moore algorithm for fast pattern matching. If you think that you have enough experience and knowledge and want to get your hands dirty, please continue to section 7 (Implementation).

A mission critical system is totally different from an office application. Imagine that your team plans to develop a firewall, or web filter, or even a secure proxy server which processes tons of packets in a few seconds. Well, what are the main characteristics of such systems?

Being fail-safe, high-performing, full-featured, and Green, are a few. By Green, I mean the system should not eat out the CPU and the memory of the hosted platform. Meanwhile, in such an expensive project, there is no room for logical mistakes due to lack of technical knowledge! In a typical environment, your last result must be deployed in the model below:

Screenshot - HttpFilter.jpg

4.1 TCP/IP protocol stack

As the name describes, TCP/IP refers to a number of protocols, each of which was developed for a purpose. In order to understand an HTTP session establishment process, we should know at-least the following protocols: Ethernet II, ARP, IP, TCP, UDP, HTTP, and DNS.

4.2 Ethernet II

The Ethernet II frame format was defined by the Ethernet specification created by Digital, Intel, and Xerox before the IEEE 802.3 Specification. The Ethernet II frame format is also known as the DIX frame format.

Ethernet II consist of the following fields (totally 26 bytes + a payload from 46 bytes to 1500 bytes):

  • Preamble (8 bytes) consisting of 7 bytes of alternating 1s and 0s (each byte is the bit sequence 10101010) to synchronize a receiving station, and a 1-byte 10101011 sequence that indicates the start of a frame. The Preamble provides receiver synchronization and frame delimitation services.
  • Destination Address (6 bytes) indicates the destination's address. The destination can be a unicast, a multicast, or Ethernet broadcast address. A unicast address is also known as an individual, physical, hardware, or MAC address. For the Ethernet broadcast address, all 48 bits are set to 1 to create the address 0xFF-FF-FF-FF-FF-FF.
  • Source Address (6 bytes) indicates the sending node's unicast address.
  • EtherType (2 bytes) indicates the upper layer protocol contained within the Ethernet frame. For an IP datagram, the field is set to 0x0800. For an ARP message, the EtherType field is set to 0x0806. For a complete list, see the References section.
  • Payload consists of a protocol data unit (PDU) of an upper layer protocol. Ethernet II can send a maximum-sized payload of 1500 bytes. Because of the Ethernet's collision detection facility, Ethernet II frames must send a minimum payload size of 46 bytes.
  • FCS (4 bytes) provides bit-level integrity verification on the bits in the Ethernet II frame using the CRC algorithm.

A typical capture shows you the fields below (FCS and Preamble are excluded):

    +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
    | destination   | source       | protocol |                   |
    | mac address   | mac address  | type     | IP DATAGRAM       |
    | (6 bytes)     | (6 bytes)    | 0X0800   |                   |
    +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

4.3 ARP

ARP is a broadcast-based, request-reply protocol that provides a dynamic address resolution facility to map the next-hop IP addresses to their corresponding MAC addresses.

There are two facts regarding the datalink layer that show we need ARP:

  1. When an Ethernet frame is sent from one host on a LAN to another, it is the 48-bit Ethernet address that determines for which interface the frame is destined. The device driver software never looks at the destination IP address in the IP datagram.
  2. The next-hop IP address is not necessarily the same as the destination IP address of the IP datagram. The result of the route determination process for every outgoing IP datagram is a next-hop interface and a next-hop IP address. For direct deliveries to destinations on the same subnet, the next-hop IP address is the datagram's destination IP address. For indirect deliveries to remote destinations, the next-hop IP address is the IP address of a router on the same Subnet as the forwarding host. To get that device as a next hop, the packet needs its hardware address.

4.4 IP

IP, the heart of the TCP/IP protocol suite, provides connectionless, unreliable delivery of data. By unreliable, we mean that there is no guarantee that a datagram successfully gets to its destination. By connectionless, we mean that the IP doesn't maintain any information regarding successive datagrams. On the other hand, each datagram is handled independently. IP makes the best effort to deliver packets to the next hop or the final destination. End-to-end reliability is the responsibility of upper layer protocols such as TCP.

An IP header contains the following fields which we need to know for packet processing later:

  • Version (4 bytes): Indicates the format of the Internet header.
  • IHL (4 bytes): Is the length of the IP header in 32 bit words.
  • Type of service or TOS (1 byte)

As the name indicates, it specifies how much the IP packet is important for us. Some intermediate devices evaluate this field in the case of high loads and prioritizes the datagram. In RFC 791 (Internet Protocol), this field is structured as follows:

     0     1     2     3     4     5     6     7

      +-----+-----+-----+-----+-----+-----+-----+-----+
      |                 |     |     |     |     |     |
      |   PRECEDENCE    |  D  |  T  |  R  |  0  |  0  |
      |                 |     |     |     |     |     |
      +-----+-----+-----+-----+-----+-----+-----+-----+
  D >> Delay
  T    >> Throughput
  R    >> Reliability
  Bit6 and Bit7    >> reserved
  • Total length (2 bytes):
  • The total size of the header + the payload. The total length field, conceptually, allows the length of a datagram to be up to 65,535 bytes, although such a long packet is impractical for most hosts and network devices. Later, we study what a MTU is and how it may help us to put this in reality.

    Anyway, remember that the IP header is only 20 bytes, and if there are any options, the length can go as high as 60 bytes. No more!

  • ID (2 bytes):
  • Assigned by the sender so that the receiver can decrement the fragmented IP packets due to the MTU value.

    For the second time, I have mentioned "MTU", so let us see what the MTU is. Most of the data you generate while, for example, surfing the web, are bulk data. It means that the size of data is big. The underlying media access protocol splits the bulk to smaller parts so that it can send them seamlessly over the network infrastructure. In the case of the HTTP 802.3 Ethernet Protocol, the maximum size of a datagram is 1500 bytes. This number is called MTU, and stands for Maximum Transmission Unit. In case you want to transmit 15000 bytes of data to your mate, the protocol stack splits your message to 10 * 1500 bytes and transmits them one by one. If we put 20 bytes for the IP header, 1480 byes remain for the transport layer header and the payload. This is where ID comes into picture. The protocol stack splits the message to 10 smaller messages and assigns a unique ID in the IP header. When the receiver gets all the pieces, it can do further processing over the whole message.

  • Flags (3 bits):
  • Says if the datagram is a part of the fragmented data or not.

             0   1   2
            +---+---+---+
            |   | D | M |
            | 0 | F | F |
            +---+---+---+
  • Fragment Offset (13 bits):
  • An 8-byte chunk of data is called a fragment block. The number in the Fragment Offset field reports the size of the offset in fragment blocks. The Fragment Offset field is 13 bits long, so offsets can range from 0 to 8191 fragment blocks—corresponding to offsets from 0 to 65,528 bytes.

  • TTL (1 byte):
  • This field says how long a datagram could remain alive in a network system. It measures time in seconds.

  • Protocol (1 byte):
  • Indicates the upper layer protocol type. For example:

    1 >> ICMP
    2 >> GIMP
    4 >> IP in IP encapsulation
    6 >> TCP
    17 >> ADP
    41 >> IPV6
    47 >> Generic routing encapsulation (ARE)
    50 >> IP security encapsulation security payload (ESP)
    51 >> IP security authentication header (AH)
    89 >> ASP
  • Header checksum (2 bytes):
  • To measure the integrity of the header, the protocol stack handler performs a CRC on the header and compares it with the checksum value. It is a kind of sanity check.

  • Source IP Address and Destination IP Address (each 4 bytes).
  • Options (variable length):
  • Maintains a list of optional information for the datagram.

       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |Version|  IHL  |Type of Service|          Total Length         |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |         Identification        |Flags|      Fragment Offset    |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |  Time to Live |    Protocol   |         Header Checksum       |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |                       Source Address                          |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |                    Destination Address                        |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |                    Options                    |    Padding    |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 

4.5 TCP

TCP is a fully formed Transport Layer protocol that provides a reliable data-transfer service and a method to pass TCP-encapsulated data to the Application Layer protocol. TCP has the following characteristics:

  • Connection oriented: Before any send or receive, both sides of data transfer must negotiate a TCP connection using a TCP three-way handshake process. TCP connections are terminated through a graceful TCP 4-way termination process.
  • Full duplex: From one pipe, data can be sent and received simultaneously. TCP does this job perfectly, gaining from the sequence number and acknowledgment number fields in its header.
  • Reliable: Data sent on a TCP connection is sequenced, and a positive acknowledgment is expected from the receiver. If no acknowledgment is received, the segment is retransmitted.
  • Flow control: To avoid sending too much data at one time and congesting the routers of the IP inter-network, TCP implements sender-side flow control that gradually scales the amount of data sent at one time. To avoid having the sender send data that the receiver cannot buffer, TCP implements a receiver-side flow control that indicates the amount of space left in the receiver's buffer.
  • Segmentation of application layer data: TCP segments data obtained from the Application Layer process so that it will fit within an IP datagram sent on the Network Interface Layer link. TCP peers exchange the maximum-sized segment that each can receive, and adjust the TCP maximum segment size using a Path Maximum Transmission Unit (PMTU) discovery.
  • One to one delivery of data: TCP connections are a logical point-to-point circuit between two Application Layer protocols. TCP does not provide a one-to-many delivery service.

The figure below shows a TCP segment encapsulated in an IP datagram:

<--------------- IP datagram ------------------>
++++++++++++++++++++++++++++++++++++++++++++++++
|   IP    |  TCP    |          TCP             |
| Header  | Header  |         Data             |
++++++++++++++++++++++++++++++++++++++++++++++++
20 bytes  20 bytes
At-least  At-least

TCP header:

   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |          Source Port          |       Destination Port        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                        Sequence Number                        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                    Acknowledgment Number                      |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |  Data |           |U|A|P|R|S|F|                               |
   | Offset| Reserved  |R|C|S|S|Y|I|            Window             |
   |       |           |G|K|H|T|N|N|                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           Checksum            |         Urgent Pointer        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                    Options                    |    Padding    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                             data                              |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  • Source Port (2 bytes): The source port number.
  • Destination Port (2 bytes): The destination port number.
  • Sequence Number (4 bytes): Indicates the outgoing byte-stream-based sequence number of the segment's first octet.
  • Acknowledgment Number (4 bytes): If the ACK control bit is set, this field indicates the sequence number of the next octet in the incoming byte stream that the receiver expects to receive.
  • Data Offset (4 bits): Indicates where the TCP segment data begins. The Data Offset field is also the TCP header's size. Just as in the IP header's Header Length field, the Data Offset field is the number of 32-bit words (4-byte blocks) in the TCP header. For the smallest TCP header (no options), the Data Offset field is set to 5 (0x5), indicating that the segment data begins in the twentieth octet offset starting from the beginning of the TCP segment (the offset starts its count at 0). With a Data Offset field set to its maximum value of 15 (0xF), the largest TCP header, including TCP options, can be 60 bytes long.
  • Reserved (6 bits): Reserved for future use. Must be zero.
  • Control Bits (6 bits from left to right):
    • URG: Urgent pointer field significant
    • ACK: Acknowledgment field significant
    • PSH: Push function
    • RST: Reset the connection
    • SYN: Synchronize sequence numbers
    • FIN: No more data from sender
  • Window (2 bytes): The number of data octets beginning with the one indicated in the acknowledgment field which the sender of this segment is willing to accept.
  • Checksum (2 bytes): The checksum field is the 16 bit one's complement of the one's complement sum of all 16 bit words in the header and text. While computing the checksum, the checksum field itself is replaced with zeros.
  • To compute a TCP checksum, we need to consider a TCP pseudo header. The TCP pseudo header is used to associate the TCP segment with the IP header. The TCP pseudo header is added to the beginning of the TCP segment only during the checksum calculation, and is not sent as part of the TCP segment. The use of the TCP pseudo header assures the receiver that a routing or fragmentation process did not improperly modify the key fields in the IP header.

    +--------+--------+--------+--------+
    |           Source Address          |
    +--------+--------+--------+--------+
    |         Destination Address       |
    +--------+--------+--------+--------+
    |  zero  |PROTOCOL|    TCP Length   |
    +--------+--------+--------+--------+
  • Urgent Pointer (2 bytes): This field communicates the current value of the urgent pointer as a positive offset from the sequence number in this segment. The Urgent Pointer points to the sequence number of the octet following the urgent data. This field is only interpreted in segments with the URG control bit set.
  • Options (variable): Options may occupy space at the end of the TCP header, and are a multiple of 8 bits in length.
  • Padding (variable): The TCP header padding is used to ensure that the TCP header ends and data begins on a 32 bit boundary. The padding is composed of zeros.

4.6 UDP

UDP is a simple, datagram-oriented, transport layer protocol: each output operation by a process produces exactly one UDP datagram, which causes one IP datagram to be sent. This is different from a stream-oriented protocol such as TCP, where the amount of data written by an application may have little relationship to what actually gets sent in a single IP datagram.

+--------+--------+--------+--------+  ----
|     Source   2  | Destination   2 |    |
|      Port  bytes|    Port    bytes|    |
+--------+--------+--------+--------+  8 bytes
|     2 bytes     |     2 bytes     |    |
|     Length      |    Checksum     |    |
+--------+--------+--------+--------+  ----
|                                   |  
|          data octets              |
+-----------------------------------+

4.7 DNS

The Domain Name System, or DNS, is a distributed database that is used by TCP/IP applications to map between hostnames and IP addresses, and to provide electronic mail routing information. We use the term distributed because no single site on the Internet knows all the information. Each site (university department, campus, company, or department within a company, for example) maintains its own database of information, and runs a server program that other systems across the Internet (clients) can query. The DNS provides the protocol that allows clients and servers to communicate with each other.

From an application's point of view, access to the DNS is through a resolver. On Windows and Unix hosts, the resolver is accessed primarily through two library functions, gethostbyname() and gethostbyaddr(), which are linked with the application when the application is built. The first takes a hostname and returns an IP address, and the second takes an IP address and looks up a hostname. The resolver contacts one or more name servers to do the mapping.

Note that DNS is an application layer protocol which is implemented over the UDP transport protocol.

4.8 HTTP

Simply put, HTTP is the protocol behind the World Wide Web. With every web transaction, HTTP is invoked.

HTTP headers

There are four types of header for HTTP:

  • General headers indicate general information such as the date, or whether the connection should be maintained. They are used by both clients and servers.
  • Request headers are used only for client requests. They convey the client's configuration and desired document format to the server.
  • Response headers are used only in server responses. They describe the server's configuration and information about the requested URL.
  • Entity headers describe the document format of the data being sent between a client and a server.

Although Entity headers are most commonly used by the server when returning a requested document, they are also used by clients when using the POST or PUT methods.

Client requests

  • GET is used to retrieve a resource on the server.
  • HEAD is used to retrieve some information about the document, but don't need the document itself.
  • POST says that you are providing some information of your own (i.e., in forms). This may change the state of the server in some way, such as creating a record in a database.
  • PUT is used to replace or create a new document on the server.
  • DELETE is used to remove a document on the server.
  • TRACE is used for protocol debugging purposes.
  • OPTIONS is used when the client looks for other methods which can be used on the document.
  • CONNECT is used when a client needs to talk to an HTTPS server through a proxy server.

Other HTTP methods that you may see (LINK, UNLINK, and PATCH) are less clearly defined.

For the reader of this article, only an understanding of the GET method is needed. This is the main method used for document retrieval. The response to a GET request can be generated by the server in many ways.

After the client uses the GET method in its request, the server responds with a status line, headers, and data requested by the client. If the server cannot process the request, due to an error or lack of authorization, the server usually sends an explanation in the entity-body of the response.

For example:

A client request

GET / HTTP/1.1\r\n
Accept: */*\r\n
Referer: http://www.google.com\r\n
Accept-Language: en-us\r\n
UA-CPU: x86\r\n
Accept-Encoding: gzip, deflate\r\n
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; SV1; 
                         .NET CLR 1.1.4322)\r\n
Host: www.google.com\r\n
Connection: Keep-Alive\r\n
Cache-Control: no-cache\r\n
Cookie: PREF=ID=76325601198c7def:LD=en:CR=2:TM=1184565890:
              LM=1186242386:S=ofiC-IBOMCVySJCP\r\n
\r\n

The server response

HTTP/1.0 200 OK\r\n
Cache-Control: private\r\n
Content-Type: text/html; charset=UTF-8\r\n
Content-Encoding: gzip\r\n
Server: GWS/2.1\r\n
Content-Length: 2976\r\n
Date: Sat, 04 Aug 2007 15:46:45 GMT\r\n
X-Cache: MISS from netcache-1\r\n
Connection: keep-alive\r\n
\r\n

4.9 Performing pattern search by using the Boyer-Moore algorithm

The Boyer-Moore approach is very interesting. It doesn't need to actually check every character of the string to be searched, but rather skips over some of them. Let's say we want to find a word "ROTTEN" in a string like HTTP://WWW.WEBSERVER.COM/ROTTEN HTTP/1.1. Using the Boyer-Moore algorithm, we have the following steps:

HTTP://WWW.WEBSERVER.COM/ROTTEN HTTP/1.1

ROTTEN
  • First, we fetch "/" because it stands in the same position as the last character of the pattern.
  • There is no "/" in the pattern. So the search fails.
  • Move the pattern right along 6 characters. So 'N' stands under 'W'. Match fails because there's no 'W' either.
  • Move the pattern right along 6 characters. So 'N' stands under 'V'. Match fails because there's no 'V' either. Also, for 'M', a similar state will happen.
  • Move the pattern right along 6 characters. So 'N' stands under 'E'. A match is found. There is an 'E' in the pattern. So, move the pattern right along a single place so both 'E's line up. And, compare right to left until the exact match is found.

This is exactly what we want. I had a comparison between Boyer-Moore and other case insensitive pattern matching methods. The results were much more efficient when I used Boyer-Moore. For more information about the Boyer-Moore pattern matching algorithm, please Google the topic.

5. An HTTP request life-time

In this section, we going to describe what happens until a page is downloaded from the Internet. Say, you open up your favorite Web browser, and type http://www.google.com in your address bar. Again, remember our network model.

Please run Ethereal, select your outgoing interface, and start capturing. You can check for all the information below, in Ethereal.

  1. The browser tries to resolve the address to its corresponding IP address. To do this, it makes a single function call to gethostbyname(). This function generates a DNS query. A DNS query, as described in section 4.7, is a UDP datagram which is destined to the DNS server which is set in the TCP/IP properties. The TCP/IP handler takes a look at the destination IP address and realizes that the client IP address and the server IP address are in different subnets, so decides to forward the packet to the next hop, which is our gateway. To do this, the driver looks for the gateway's MAC address in the local ARP table. If, by any reason, the gateway's MAC address is not among the resolved address, TCP/IP broadcasts an ARP request in the subnet to get the corresponding MAC address. Upon receiving the gateway's MAC address, the client makes the final packet and puts it on the wire. Then, the packet is received by the gateway. The gateway looks at the destination IP address and realizes that this packet doesn't belong to itself, so decides to forward this to the next hop. From now, the packet traverses a global route until it successfully gets to the destination. The destination protocol handler then hands away the packet to the proper application layer service, which is the DNS in this case. Finally, the DNS server fetches the related IP address and makes a DNS response destined to the sender.
  2. Destination IP address in hand, our client protocol stack can now establish a TCP session with the peer.
  3. The "three-way handshake" is the procedure used to establish a connection. This procedure, normally, is initiated by one TCP and responded to by another TCP.

    The figure below shows the exact process in our example:

          You                                                         web server
    
    1.  CLOSED                                                       LISTEN
    
    2.  SYN-SENT    --> <SEQ=ISN1><CTL=SYN><ACK=0>               --> SYN-RECEIVED
    
    3.  ESTABLISHED <-- <SEQ=ISN2><ACK=ISN1+1><CTL=SYN,ACK>      <-- SYN-RECEIVED
    
    4.  ESTABLISHED --> <SEQ=ISN1+1><ACK=ISN2+1><CTL=ACK>        --> ESTABLISHED
    
    5.  ESTABLISHED --> <SEQ=ISN1+1><ACK=ISN2+1><CTL=ACK><DATA>  --> ESTABLISHED

    In line 2 of the figure, you begin by sending a SYN segment indicating that you will use sequence numbers starting with the sequence number ISN1. In line 3, the server sends a SYN and acknowledges the SYN it received from you. Note that the acknowledgment field indicates that the server is now expecting to hear the sequence ISN1+1, acknowledging the SYN which occupied the sequence ISN1. At line 4, you respond with an empty segment containing an ACK for the server's SYN.

  4. The application layer transactions begins.
  5. In line 5, you send some data. Note that the sequence number of the segment in line 5 is the same as in line 4 because the ACK does not occupy the sequence number space. As a result, <DATA> contains the following statements:

    GET / HTTP/1.1\r\n
    Accept: */*\r\n
    Referer: http://www.google.com\r\n
    Accept-Language: en-us\r\n
    UA-CPU: x86\r\n
    Accept-Encoding: gzip, deflate\r\n
    User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; SV1; 
                             .NET CLR 1.1.4322)\r\n
    Host: www.google.com\r\n
    Connection: Keep-Alive\r\n
    Cache-Control: no-cache\r\n
    Cookie: PREF=ID=76325601198c7def:LD=en:CR=2:
              TM=1184565890:LM=1186242386:S=ofiC-IBOMCVySJCP\r\n
    \r\n
  6. The server receives the client's request and responds back with some data.
  7. Finishing data transfer. To do this, TCP provides you with two choices: TCP reset and TCP graceful termination.
  8. TCP resets are usually sent when something goes wrong ether for the client or the server. To see a real TCP reset, simply type http://www.codeproject.com in your address bar and press the Esc button after a few moments. A TCP reset packet is a 40 byte datagram with the TCP flags set to 0x0004 (RST). On the other hand, we have a graceful connection termination which is a 4-way sequence.

             You                                               web server
    
      1.  ESTABLISHED                                          ESTABLISHED
    
      2.  (Close)
          FIN-WAIT-1  --> <SEQ=ISN1><ACK=ISN2><CTL=FIN,ACK>    --> CLOSE-WAIT
    
      3.  FIN-WAIT-2  <-- <SEQ=ISN2><ACK=ISN1+1><CTL=ACK>      <-- CLOSE-WAIT
    
      4.                                                       (Close)
          TIME-WAIT   <-- <SEQ=ISN2><ACK=ISN1+1><CTL=FIN,ACK>  <-- LAST-ACK
    
      5.  TIME-WAIT   --> <SEQ=ISN1+1><ACK=ISN2+1><CTL=ACK>    --> CLOSED
    
      6.  (2 MSL)
          CLOSED

    Note that the TCP connection terminations do not have to use four segments. In some cases, segments 2 and 3 are combined. The result is a FIN-ACK/FIN-ACK/ACK sequence.

6. Our solution for implementing a web filter

The solution is fairly simple. The following sequence is used to filter a matched web request:

  1. Capture packets.
  2. Distinguish TCP packets which are destined to port 80 or 8080.
  3. Look for a 'GET /' in the first four bytes of the data. If not found, continue with the next captured packet.
  4. Perform a Boyer-Moore pattern match over the data against the blacklist keywords. If not found, continue with the next captured packet.
  5. Send a TCP reset to the server and a block page to the client.

7. Implementation

This section explains the core functionality of our HTTP filter in details. All functionalities are available in the source code in the CProcessPacket class.

7.1 Capturing packets with a raw socket

The fastest way to capture incoming traffic without using a NDIS low level driver is to use a Winsock raw socket. Although, you should note that a raw socket cannot capture in promiscuous mode. This means that the socket captures only traffic that is destined to its own address.

if((m_sniffSocket = socket(AF_INET, SOCK_RAW, IPPROTO_IP))==SOCKET_ERROR)
{        
    return 0;
}
PIP_ADAPTER_INFO pAdapterInfo = m_AdapterInfo; // Adapter information

u_long in = 0;
do {
    if (strcmp (in_szSourceDevice, pAdapterInfo->AdapterName ) == 0)
    {
        break; // we found selected adapter

    }
    in++;
    pAdapterInfo = pAdapterInfo->Next;    // Progress through
}
while(pAdapterInfo);  
        

struct sockaddr_in src;
memset(&src, 0, sizeof(src));
src.sin_addr.S_un.S_addr = 
   inet_addr (pAdapterInfo->IpAddressList.IpAddress.String);        
src.sin_family      = AF_INET;
src.sin_port        = 0;
if (bind(m_sniffSocket,(struct sockaddr *)&src,sizeof(src)) == SOCKET_ERROR)
{            
    return 0;
}

int j=1;
if (WSAIoctl(m_sniffSocket, SIO_RCVALL, &j, 
             sizeof(j), 0, 0, &in,0, 0) == SOCKET_ERROR)
{            
    return 0;
}

And, then, in a separate thread, we process the captured packets:

    int  res = 0;
    char *pkt_data = (char *)malloc(65536); 
    char                    m_pLogString[256];        
    Packet                    p;
    
    if (pkt_data == NULL)
    {        
        return 0;
    }

    SetEvent(_this->m_hThrdReadyEvent);
    
    // Capture packet

    do
    {
        res = recvfrom(_this->m_sniffSocket,pkt_data,65536,0,0,0); 
        if(res > 0)    
        {            
            ZeroMemory(&p, sizeof (Packet));
            
            DecodeIP((u_int8_t*)pkt_data, res, &p);

            if (p.banned == 1)
            {
                FilterHttpRequest(&p);

                char ip_string_src[17];
                char ip_string_dst[17];

                memcpy (ip_string_src, inet_ntoa(p.iph->ip_src), 17);
                memcpy (ip_string_dst, inet_ntoa(p.iph->ip_dst), 17);

                sprintf (m_pLogString,
                         "Keyword \'%s\' is detected in a request" 
                         " from %s to %s. Session Droped!", 
                         p.matched, ip_string_src, ip_string_dst);

                m_pFilterLog->AddLog(m_pLogString);
                
            }

        }
    } 
    while (res > 0);

7.2 Distinguishing HTTP packets

In this section, we process the portions of the captured packet to find the Ethernet, IP, and TCP header fields. To perform this, we need a couple of handy structs to lay data over them.

struct IPHdr
{
    u_int8_t ip_verhl;      /* version & header length */
    u_int8_t ip_tos;        /* type of service */
    u_int16_t ip_len;       /* datagram length */
    u_int16_t ip_id;        /* identification  */
    u_int16_t ip_off;       /* fragment offset */
    u_int8_t ip_ttl;        /* time to live field */
    u_int8_t ip_proto;      /* datagram protocol */
    u_int16_t ip_csum;      /* checksum */
    struct in_addr ip_src;  /* source IP */
    struct in_addr ip_dst;  /* dest IP */
};

struct TCPHdr
{
    u_int16_t th_sport;     /* source port */
    u_int16_t th_dport;     /* destination port */
    u_int32_t th_seq;       /* sequence number */
    u_int32_t th_ack;       /* acknowledgement number */
    u_int8_t th_offx2;     /* offset and reserved */
    u_int8_t th_flags;
#define    TH_FIN    0x01
#define    TH_SYN    0x02
#define    TH_RST    0x04
#define    TH_PSH    0x08
#define    TH_ACK    0x10
#define    TH_URG    0x20
    u_int16_t th_win;       /* window */
    u_int16_t th_sum;       /* checksum */
    u_int16_t th_urp;       /* urgent pointer */
};

// our helper struct to hold packet information in one place

struct Packet
{
    
    u_int8_t            *pkt;    /* base pointer to the raw packet data */
    
    
    IPHdr                *iph;   /* and orig. headers for ICMP_*_UNREACH family */
    u_int32_t            ip_options_len;
    u_int8_t            *ip_options_data;


    TCPHdr                *tcph;
    u_int32_t            tcp_options_len;
    u_int8_t            *tcp_options_data;


    u_int8_t            *data;            /* packet payload pointer */
    u_int16_t            dsize;           /* packet payload size */

    
    
    u_int8_t            http_state;       /* HTTP request / HTTP response */
    u_int8_t            banned;           /* Indicate if the request should be sensored */
    unsigned char        matched[128];    /* Keyword that this request 
                                             matched to - maximum 128 byte*/
    
#define    CLIENT_REQUEST    0x01
#define    SERVER_RESPONSE    0x02
#define    NOT_HTTP        0x04

    
    u_int8_t frag_flag;     /* flag to indicate a fragmented packet */
    u_int16_t frag_offset;  /* fragment offset number */
    u_int8_t mf;            /* more fragments flag */
    u_int8_t df;            /* don't fragment flag */
    u_int8_t rf;            /* IP reserved bit */
};

Now, we can retrieve all the header information stored in the captured packet. To perform this step, you should know the related protocol specifications which we explained in section 4.

Packet p;
    
/* lay the IP struct over the raw data */
p->iph = (IPHdr *) (pkt_data); 
    
/* Decode only TCP headers */
if (p->iph->ip_proto == 6)
{
    /* lay TCP on top of the data cause there is enough of it! */
    p->tcph = (TCPHdr *) (pkt_data + IP_HEADER_LEN);
    // set pinter at the beginning of TCP header 

    if (p->tcph->th_flags & TH_ACK /* If is request */
        && p->tcph->th_flags & TH_PSH)
         {
             if(p->tcph->th_dport != htons(80) &&
                /* If target service is not HTTP */
                p->tcph->th_dport != htons(8080) )
             return ;
             // continue in 7-3 }

         }
}

7.3 Looking for 'GET /' in the first 4 bytes

p->data = (unsigned byte*)(pkt_data + ETHERNET_HEADER_LEN + 
           IP_HEADER_LEN + TCP_HEADER_LEN)
// find payload position


if( p->data[0] == 'G' &&
        p->data[1] == 'E' &&
        p->data[2] == 'T' &&
        p->data[3] == ' ' &&
        p->data[4] == '/' 
        )
{
    // continue in 7-4

}

7.4 Performing a Boyer-Moore pattern matching algorithm on the payload

As I explained earlier, the Boyer-Moore is fast pattern matching algorithm which follows a non-liner approach to find a match.

if (CheckPattern(p->data, len))
{
    // A match found, session must be filtered!
    // continue in 7-5
}

7.5 Finishing the data transaction by sending a bunch of packets to both directions

Perhaps, this section is the most tricky part of the article. Here, I again remind that our HTTP filter works in Sniffer mode. As a result, we should do our best to prevent our reliable TCP engines on both sides liven up the session by resynchronizing.

  1. Let's say we grasp a target HTTP GET packet with the following TCP information:
  2. SEQ = ISN1 ACK = ISN2 
    Direction = To Server
  3. Again, due to the fact that we are operating in a monitor mode, we should get rid of server resynchronization attempts. For this purpose, I sent a TCP reset to the server to stop the TCP state machine for a while.
  4. SEQ = ISN1 ACK = ISN2
    FLAGS = RST 
    Direction = To Server.
  5. Sending a block page to the client:
  6. SEQ = ISN2 ACK = ISN1 + Len(GET)
    FLAGS = FIN | ACK
    Direction = To Client.

    Note that the implemented routine is available in CProcessPacket::SensorHttpRequest.

8. Special consideration

While it's a choice for the SOHO environment, our HTTP filter is a less common type of filtering solution for enterprise networks. This comes from the fact that the sniffing mode doesn't guarantee a synchronized reaction against TCP sessions because it doesn't stand against packet flows. Compare it with a football stadium before and after a game. Before the game, guards can inspect fans one by one and check their tickets. After the game, fans just rush outdoors, and it is fairly easy for an insurgent to get out of sight. At least, in my city, it's this way! Wink | ;)

It's a matter of one-by-one inspection! Along with its reliability, a one-by-one inspection has a big flaw: "a slow inspector could slow down the whole movement".

9. Future work

  • Make the concept more reliable by performing asynchronous analysis in background threads
  • Replace the core capturing functionalities with a network driver hook available in both Windows and Unix platforms
  • Replacing the Boyer-Moore with a multi-pattern search so that dozens of patterns are evaluated in a single search
  • Look for your ideas or requests

10. Revision history

  • Initial release - 2007-08-06
  • Code comments improved - 2007-08-07
  • Total review; Excluding Winpcap and Libnet from the project; Performing both capture and send raw packets by using the raw socket functionality available in WinSock - 2007-08-26

11. My test environments

  • Ethernet LAN
  • WLAN 802.11/bg standard sub-network

Anyway, if you experience any malfunction, please report it. Thanks in advance!

12. References

  1. TCP/IP Illustrated, Volume 1 The Protocols W. Richard Stevens
  2. RFC 793, RFC 791, RFC 768, RFC 826, RFC 1034, RFC 1035. Refer to IETF
  3. Network programming and network security programming forum
  4. Ethereal
  5. Winpcap
  6. Windows server 2003 TCP/IP and services. Technical reference by Josef Davies and Thomas Lee.
  7. HTTP pocket reference by Clinton Wong, O'Reilly May 2000.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

eternalnetworker
Architect
Australia Australia
http://www.linkedin.com/pub/reza-shadmani/56/812/875/

Comments and Discussions

 
QuestionHTTP Filter Question PinmemberDanCheshire14-Feb-12 8:39 
QuestionEngine Won't Start PinmemberMember 77661801-Oct-11 12:54 
AnswerRe: Engine Won't Start Pinmembereternalnetworker19-Dec-11 20:45 
Questionquery about this module Pinmembervaibhavbvp30-Jun-11 23:30 
AnswerRe: query about this module Pinmembereternalnetworker1-Jul-11 19:01 
GeneralRe: query about this module Pinmembervaibhavbvp1-Jul-11 23:25 
GeneralRe: query about this module Pinmembereternalnetworker19-Dec-11 20:42 
Generalsite packets Pinmemberabdallah22233325-Oct-09 21:38 
Generalincoming packets Pinmemberabdallah22233325-Oct-09 13:38 
GeneralRe: incoming packets Pinmembereternalnetworker25-Oct-09 20:48 
GeneralXP Pinmemberradgoate19-Oct-09 5:19 
GeneralRe: XP Pinmembereternalnetworker25-Oct-09 20:51 
GeneralRe: XP PinmemberZUPERKOOL26-Jan-10 9:31 
GeneralContent Filter PinmemberAseem Sharma13-Oct-09 14:32 
GeneralRe: Content Filter Pinmembereternalnetworker17-Oct-09 21:37 
GeneralRe: Content Filter PinmemberAseem Sharma24-Oct-09 4:59 
GeneralRe: Content Filter Pinmembereternalnetworker25-Oct-09 21:02 
GeneralRe: Content Filter [modified] PinmemberAseem Sharma26-Oct-09 17:05 
GeneralRe: Content Filter PinmemberAseem Sharma13-Nov-09 5:07 
GeneralRe: Content Filter Pinmembereternalnetworker17-Nov-09 22:56 
GeneralRe: Content Filter Pinmembereternalnetworker17-Nov-09 22:57 
GeneralRe: Content Filter Pinmembereternalnetworker30-Nov-09 3:04 
GeneralRe: Content Filter PinmemberAseem Sharma22-Dec-09 20:24 
Generalstop loading page Pinmemberjasaskripsi24-Nov-08 19:23 
GeneralRe: stop loading page Pinmembereternalnetworker11-Dec-08 18:49 
Generalhelp Pinmemberlipozom14-Oct-08 6:44 
GeneralRe: help Pinmembereternalnetworker18-Oct-08 6:22 
GeneralRe: help Pinmembermydearzct11-Mar-09 18:53 
GeneralRe: help PinmemberPrashant Gohil16-Mar-09 20:29 
GeneralRe: help PinmemberSt.Devil14-Sep-10 17:32 
GeneralThis cannot be used to bypass proxy [modified] Pinmemberthe pink jedi4-Apr-08 22:56 
GeneralRe: This cannot be used to bypass proxy Pinmembereternalnetworker5-Apr-08 5:45 
GeneralRe: This cannot be used to bypass proxy Pinmemberthe pink jedi5-Apr-08 18:25 
GeneralRe: This cannot be used to bypass proxy Pinmembereternalnetworker6-Apr-08 5:16 
GeneralRe: This cannot be used to bypass proxy Pinmembereternalnetworker6-Apr-08 5:20 
Questionabout get http request packet in inside ethernet card Pinmembersunliangxu8-Mar-08 3:46 
GeneralRe: about get http request packet in inside ethernet card Pinmembereternalnetworker13-Mar-08 8:04 
GeneralRe: about get http request packet in inside ethernet card Pinmembersunliangxu16-Mar-08 3:06 
GeneralRe: about get http request packet in inside ethernet card Pinmembereternalnetworker18-Mar-08 23:31 
GeneralRe: about get http request packet in inside ethernet card Pinmembersunliangxu30-Mar-08 14:56 
GeneralWindows server 2003 is the only working environment for this article Pinmembereternalnetworker15-Jan-08 6:31 
QuestionProblems on Windows XP SP2 Pinmembermandragorax5-Jan-08 4:41 
GeneralRe: Problems on Windows XP SP2 Pinmembereternalnetworker5-Jan-08 6:06 
GeneralRe: Problems on Windows XP SP2 Pinmembermandragorax5-Jan-08 9:03 
GeneralRe: Problems on Windows XP SP2 Pinmembermandragorax6-Jan-08 0:32 
GeneralRe: Problems on Windows XP SP2 Pinmembermandragorax6-Jan-08 0:39 
GeneralRe: Problems on Windows XP SP2 Pinmembereternalnetworker15-Jan-08 6:29 
AnswerRe: Problems on Windows XP SP2 Pinmemberdensie30-Jul-08 20:53 
Questionchinese characters Pinmemberhhyjmd19-Nov-07 20:53 
AnswerRe: chinese characters Pinmembereternalnetworker22-Nov-07 5:21 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Mobile
Web01 | 2.8.141015.1 | Last Updated 26 Aug 2007
Article Copyright 2007 by eternalnetworker
Everything else Copyright © CodeProject, 1999-2014
Terms of Service
Layout: fixed | fluid