Retrieve Size of Data Available C# TCP NetworkStream

Question

0.00/5 (No votes)

See more:

, +

I would like to be able to get the length of the data available from a tcp network stream in C# to set the size of the buffer before reading from the network stream. There is a NetworkStream.Length property but it isn't implemented yet, and I don't want to allocate an enormous size for the buffer as it would take up too much space. The only way I though of doing it would be to precede the data transfer with another telling the size, but this seems a little messy. What would be the best way for me to go about doing this.

-Jordan

Posted 15-Oct-13 8:07am

Sicppy

Add a Solution

Comments

Sergey Alexandrovich Kryukov 15-Oct-13 14:40pm

Why "messy"? This is the best thing you could think of.
—SA

2 solutions

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Sergey Alexandrovich Kryukov · Accepted Answer · 2013-10-15T08:39:00

Solution 1

It's not that this property is not "yet" implemented; such implementation would not make any sense. Think about the nature of network streams: a remote part writing to the stream can always write some more data. Also note that the stream is the abstraction behind the sequence of TCP packets, so, even though you can determine the size of the package, it does not tell you the expected size of all of the data, because you don't know in advance if all the data is even sent in full or not. Actually, this is a matter of some application-layer protocol, which always present on top of some transport-layer protocol (TCP or any other). Even you did not define such protocol explicitly for your applications (but you should better do it), it always exist implicitly. So, such protocol may or may not assume some finite stream size. In cases different from yours, it may assume "infinite" exchange of small messages.

So, what to do? Actually, you already had the perfect idea: "precede the data transfer with another telling the size". It's not "messy" at all, just to opposite, this is the best thing you can do. You should consciously include this size in your application-layer protocol, formally and accurately. It does not have to be "low-level" size in bytes, it could be, say, something like "number of sent elements". And then, if each element also can have variable size, you can preface each element with its own size, and in case of compose object, do it hierarchically. This is how, in particular, binary serializers work. So, one optional technique would be using a binary serializer for your data.

—SA

Posted 15-Oct-13 8:39am

Sergey Alexandrovich Kryukov

v2

Comments

Sicppy 15-Oct-13 14:51pm

It just seemed like more work than really needed to be done. I achieved the same sort of result by reading the stream into memory in 1kb segments until the stream was empty, then I could simply read the length from the memory stream, thank you for the reply however.

Sergey Alexandrovich Kryukov 15-Oct-13 15:02pm

I don't think so. If you really need to know the size of the data of variable size, this is what you need. Your method certainly involves a lot more hassle. And in prefix approach, this is (in the simplest case), you just add reading/writing just one extra number.

You are about to make a wrong decision. Think again, I tell you.

—SA

Sicppy 15-Oct-13 15:09pm

I don't need to know the size, I just needed to be able to get the entire stream

Sergey Alexandrovich Kryukov 15-Oct-13 15:24pm

Nevertheless... I warn you: you about to make a big mistake to be sorry about later. Now, you decide.
—SA

Sicppy 15-Oct-13 15:35pm

This has worked for me perfectly fine so far, but I will try the other as well.

SoMad 15-Oct-13 15:41pm

It will probably work fine most of the time if both applications are on the same machine and possibly on a small scale LAN, but things work differently on a larger enterprise network, not to mention over the Internet.

Soren Madsen

Sicppy 15-Oct-13 15:47pm

I'm probably going to do both, precede with the size so I know how many to expect and then keep collecting in increments to ensure that I get all of the data

Sergey Alexandrovich Kryukov 15-Oct-13 15:57pm

When something is undecided, a little precaution won't hurt. :-)
—SA

Sicppy 15-Oct-13 22:33pm

Exactly!

Sicppy 15-Oct-13 23:54pm

Look at my modified code and tell me what you think, the first 20 bytes are the size, which it reads immediately and converts to a string, and then an int. Then it loops through and checks if data is available and if there is it reads the data it loaded off into a memory stream, and it will keep doing this until the amount of bytes gathered is equivalent to the amount of bytes sent.

Sergey Alexandrovich Kryukov 16-Oct-13 0:01am

Why 20 bytes for the size? This is wrong; you really want to use binary streams with TCP, and nothing prevents you from sending/receiving texts in this stream as well. Now I understand your hesitation in using prefixed data structure: if you are doing it in text, of course some additional hassle would be dealing with different lengths of the string representing length (not difficult, too, but still it would make the whole idea a bit questionable). Of course, switch to binary mode.

Using text for representing numbers is a part of more general fallacy typical for the beginners this days: working with strings representing data instead of data itself. Don't do it, try to work with data, not strings...

—SA

Sicppy 16-Oct-13 23:10pm

ASCII is the best way to represent the number I think because each character only takes up one byte, and that code has since been changed slightely. And I am a bit of a beginner to working directly with bytes, but hey, I'm only 16, I have time to learn.

Sergey Alexandrovich Kryukov 17-Oct-13 0:04am

Who told you all that gibberish? (By the way, ASCII is not an actual standard for a decade or more. Behaving like you were a mastodon of the age 40+? :-) Not only binary is easier, it's of course easier. You don't have to think about bytes: it should be obvious that, if you write something in binary, and later read from the same place, symmetrically, you will just get the same thing, no matter what it was. With strings, it's way more of a hassle. I don't even understand how can it be not obvious.

Anyway, this is not even a point. How come you are doing advanced things like network and don't want to start with basics, like binary? Want to waste time to later come back? Nothing terrible, but better be logical...

—SA

Sicppy 17-Oct-13 2:22am

How would I represent a large number without taking up a large amount of bytes though?

Sergey Alexandrovich Kryukov 17-Oct-13 3:01am

It depends on what do you mean by "represent". What "large amount"? Length is 32 or 64-bit unsigned integer, that is, 4 or 8 bytes. The number if them is sizeof(type). Can you finally understand that it has nothing to do with bytes? You just read and write the same type, without any concern about "representation" and even the size.


ulong sizeOfSomething = //...
myStream.Write(sizeOfSomething);

// 3 years passed... :-)

ulong sizeOfSomething = myStream.ReadUint64();
//...

What?...

Anyway, it's not my mission to convince you. If you are doing wrong thing, who would be the looser? :-)

—SA

Sicppy 17-Oct-13 3:08am

I'm not trying to be convinced, and I'm not asking you to convince me, I just want to know how I can get the biggest possible number into the smallest amount of bytes and be able to read it when it shows up on the other end

Sergey Alexandrovich Kryukov 17-Oct-13 3:12am

Of course you are not trying. You are trying to stay in comfort zone and not learn anything, even the obvious. Trying really hard. This is becoming boring. Isn't the answer to your biggest/smallest obvious to you?
—SA

Sicppy 17-Oct-13 3:16am

I find that comment pretty rude and completely incorrect, the whole reason I am doing this is to learn. I am trying to ask you how I might go about representing an integer in bytes, I am not ignoring you nor disregarding you, I am simply asking for your advice, is that not obvious to you?

Sicppy 17-Oct-13 3:27am

I have just found the BitConverter is that how you would suggest sending the ULong value?

Sergey Alexandrovich Kryukov 17-Oct-13 3:31am

This is a good thing, but the binary reader/writer reads/writes primitive types without the help of such serializer, it's already done. Here:
http://msdn.microsoft.com/en-us/library/system.io.binaryreader.aspx,
http://msdn.microsoft.com/en-us/library/system.io.binarywriter.aspx.

—SA

Sicppy 17-Oct-13 3:41am

Is there any way I can use those to write/read to/from a Byte[] ?

Sicppy 17-Oct-13 3:57am

Nevermind I figured that out.

Sicppy · Accepted Answer · 2013-10-15T08:47:00

I have answered my question but I'll post it here for anyone who is curious, it is impossible to read the length of the data without any secondary information being sent, however it is possible to load chunks off the stream until the stream is empty, this is what I have achieved with these pieces of code:

Send:

C#

public void SendPacket(Packet pkt)
        {
            NetworkStream Stream = Socket.GetStream();
            Byte[] Buffer = pkt.Serialize();
            Byte[] Size = new Byte[8];
            long len = Buffer.Length;
            using (MemoryStream mem = new MemoryStream(Size))
            {
                using (BinaryWriter BW = new BinaryWriter(mem))
                {
                    BW.Write(len);
                }
            }
            Byte[] Bytes = new Byte[8 + Buffer.Length];
            Array.Copy(Size, Bytes, Size.Length);
            Array.Copy(Buffer, 0, Bytes, 8, Buffer.Length);
            Stream.Write(Bytes, 0, Bytes.Length);
        }

Receive:

C#

private void StreamUpdater()
        {
            while(true){
                NetworkStream Stream = Socket.GetStream();
                if (Stream.DataAvailable)
                {
                    Byte[] Bytes = new Byte[0];
                    Byte[] buffer = new Byte[8];
                    Stream.Read(buffer, 0, 8);
                    long i;
                    using (MemoryStream mem = new MemoryStream(buffer))
                    {
                        using (BinaryReader BR = new BinaryReader(mem))
                        {
                            i = BR.ReadInt64();
                        }
                    }
                    long BiM = 0;
                    if(DataArrived != null)
                        DataArrived.Invoke(this, new TcpEventArgs(new Packet(Packet.Type.Null, null)));
                    Array.Clear(buffer, 0, 8);
                    Array.Resize(ref buffer, 1024);
                    while (BiM < i)
                    {
                        int bytesRead;
                        int BytesToRead = 1024;
                        if (BytesToRead > (i - BiM)) BytesToRead = (int)(i - BiM);
                        if ((bytesRead = Stream.Read(buffer, 0, BytesToRead)) > 0)
                        {
                            Array.Resize(ref Bytes, Bytes.Length + bytesRead);
                            Array.Copy(buffer, 0, Bytes, BiM, bytesRead);
                            BiM += bytesRead;
                            double percent = BiM / i;
                            percent = percent * 100;
                            PercentReceived = (int)percent;
                        }
                    }
                    Packet pkt = SerialMediator.Deserialize(Bytes);
                    if (DataAvailable != null)
                    {
                        DataAvailable.Invoke(this, new TcpEventArgs(pkt));
                    }
                }
            }
        }

Ignore the last four lines of code. The Packet and SerialMediator classes are classes I made to wrap up data being sent between the server and client, and the DataArrived is a custom event I created. The TcpEventArgs is a custom EventArgs class I created.