Reading chunked response from server

Question

0.00/5 (No votes)

See more:

Hello friends, i am in the middle of the dark now and i don't know what to do again. I am writing a proxy application that receives HTTP request from browser, forwards it to the server, read the response from server and send it back to the browser.

I wrote the program and everything works fine. But i discovered that whenever i receive a response from the server that has the "Transfer-Encoding: chunked" as part of the HTTP header response, the packet datas i read from server are incomplete. So, i do a google search to know what this chunked transfer encoding really means. I read many articles and i know what this chunked transfer really is. So, i create a sub to handle the reading of any header response that contains the "Transfer-Encoding: chunked" but still, my logic is not accurate enough because the packet datas i read from the server are still incomplete. So pls i need help on how i can read "CHUNKED" transfer encoding response from the server. Here is the header response i get from the server:

HTTP/1.1 200 OK
Date: Fri, 23 Nov 2012 08:45:52 GMT
Server: Apache
Transfer-Encoding: chunked
Content-Type: text/html
Connection: close

Here is the sub i created to handle the reading of any chunk response from the server:

VB

Sub chunkResponseReader(ByVal incoming As TcpClient, ByVal outgoing As TcpClient, ByVal client As NetworkStream, ByVal server As NetworkStream)
        While True
            Try
                Dim res_hdr As String = ""
                Dim agw As Integer
                Dim php(0) As Byte
                Dim chunkSize As Integer
                Dim buffer() As Byte
                While True
                    agw = server.Read(php, 0, php.Length)
                    res_hdr = res_hdr & Encoding.ASCII.GetString(php, 0, agw)
                    If res_hdr.IndexOf(vbNewLine) <> -1 Then
                        Exit While
                    End If
                    If agw = 0 Then
                        outgoing.Close()
                        incoming.Close()
                        Exit Sub
                    End If
                    received += agw
                End While
                totalTransfer = sent + received
                agw = 0
                chunkSize = CInt(convertHex(res_hdr.Substring(0, res_hdr.IndexOf(vbNewLine))))
                Dim chunkRemain As Integer = 0
                'Dim chunkBuff As Integer
                ReDim buffer(chunkSize + 2)
                While True
                    agw = server.Read(buffer, 0, buffer.Length)
                    If agw = 0 Then
                        incoming.Close()
                        outgoing.Close()
                        Exit Sub
                    End If
                    client.Write(buffer, 0, agw)
                    received += agw
                    chunkRemain = (chunkSize + 2) - agw
                    chunkSize = chunkRemain
                    If chunkRemain <= 0 Then
                        Exit While
                    End If
                End While
                msg("Chunk :: DONE!")
            Catch ex As Exception
                msg("Chunk Response Reader Sub :: " & ex.Message)
            End Try
        End While
    End Sub

Here is my convertHex function:

VB

Private Function convertHex(ByVal hex As String) As Object
       Try
           Return Integer.Parse(hex, Globalization.NumberStyles.HexNumber)
       Catch ex As Exception
           Return False
       End Try
   End Function

Pls help

Posted 22-Nov-12 21:41pm

Joeadeoye

Updated 22-Nov-12 22:12pm

v4

Add a Solution

1 solution

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Joeadeoye · Answer 1 · 2012-11-24T01:31:00

@all i later get to solve this problem myself after careful studying and analyzing the chunked data. Here is my chunkResponseReader sub maybe it may be of great help to another person.

VB

Sub chunkResponseReader(ByVal incoming As TcpClient, ByVal outgoing As TcpClient, ByVal client As NetworkStream, ByVal server As NetworkStream)
        While True
            Try
                Dim res_hdr As String = ""
                Dim agw As Integer
                Dim php(0) As Byte
                Dim chunkSize As Integer
                Dim buffer() As Byte

                '################################
                '#      READ CHUNK SIZE         #
                '################################
                While True
                    agw = server.Read(php, 0, php.Length)
                    res_hdr = res_hdr & Encoding.ASCII.GetString(php, 0, agw)
                    If res_hdr.IndexOf(vbCrLf) <> -1 Then
                        Exit While
                    End If
                    If agw = 0 Then
                        outgoing.Close()
                        incoming.Close()
                        Exit Sub
                    End If
                    received += agw
                End While
                totalTransfer = sent + received
                agw = 0
                chunkSize = CInt(convertHex(res_hdr.Substring(0, res_hdr.IndexOf(vbCrLf))))


                

                Dim chunkRemain As Integer = 0
                'Dim chunkBuff As Integer
                ReDim buffer(chunkSize - 1)
                Dim res_hd As String
                While True
                    agw = server.Read(buffer, 0, buffer.Length)
                    If agw = 0 Then
                        incoming.Close()
                        outgoing.Close()
                        Exit Sub
                    End If
                    client.Write(buffer, 0, agw)
                    res_hd = Encoding.ASCII.GetString(buffer, 0, agw)


                    

                    received += agw
                    chunkRemain = (chunkSize + 0) - agw
                    chunkSize = chunkRemain

                    If chunkRemain <= 0 Then
                        'read away the eof and CrLf
                        Dim chunkEOF As String = ""
                        Dim agws As Integer = 0
                        While True
                            agws = server.Read(php, 0, php.Length)
                            chunkEOF = chunkEOF & Encoding.ASCII.GetString(php, 0, agws)
                            If chunkEOF.IndexOf(vbCrLf) <> -1 Then
                                Exit While
                            End If
                            received += agws
                        End While
                        Exit While
                    Else
                        ReDim buffer(chunkRemain - 1)
                    End If

                End While
                msg("Chunk :: DONE!")
            Catch ex As Exception
                msg("Chunk Response Reader Sub :: " & ex.Message)
                incoming.Close()
                outgoing.Close()
                Exit While
            End Try
        End While
    End Sub

My convertHex function still remains the same.
Thanks