Click here to Skip to main content
Rate this: bad
good
Please Sign up or sign in to vote.
Hi,
I have a question about tcp data streams in c++. I understand that since a 32-bit integer is used to count the number of bytes sent in the stream there must be a limit of sending 2^32 bytes (4 GB) per stream, or am I wrong?
 
Is this a problem one must consider when using e.g. boost::asio or is the implementation managing this itself. I'm writing an application which will stream data of sizes possible lager than 4GB so I started to thinking if I should split the data into some several streams if it's size exceeds 4GB. Note, this is not for performance.
 
**edit**
The TCP protocol itself does contain an integer in order to count the number of bits sent, from what I understand that limits the underlying TCP stream itself from sending more data than 4GB. So I wonder if the implementation of ASIO has solved this issue behind the scenes or if you should keep track of it yourself.
 
The data isn't stored in memory for very long, so not being able to address it in memory isn't a problem. The data is also "generated" in small chunks so that's neither a problem for the sender.
********
 
Regards
WaZoX
Posted 7-Feb-13 6:19am
WaZoX854
Edited 7-Feb-13 8:59am
v2
Rate this: bad
good
Please Sign up or sign in to vote.

Solution 2

I wouldn't worry about TCP, that can take care of larger streams than 4GB. It does so every time I download a new Linux Distro these days, let alone when I'm playing with a few hundred hours of video footage.
Asio is another question but I seriously doubt if you'll break it with anything like that. The only way to be 100% sure though would be to try it because Asio is complex and there's more than one way to use it so even asking the Asio devs might not give you a precise answer.
To be honest I wouldn't worry about it at all unless you're already seeing a problem. The last time I went hunting for a bug in Asio it nearly melted my brain and the fault turned out to be in the embedded TCP implementation of the source the stream was coming from. I've used Asio indirectly to process ~1TB of TCP data in 12 streams that were continuous over several days while simultaneously wiresharking the whole mess and it stood up.
  Permalink  
Comments
WaZoX at 8-Feb-13 9:54am
   
Sounds great if there isn't no such limitation or if the underlying implementation takes care of that. Maybe stupid question but would you consider ASIO to be reliable enough to e.g. base the network code on it in a commercial product? I guess I gonna write a simple test application and try if it works but it sounds very promising. Thanks a lot for your answer and for sharing your experience.
Matthew Faithfull at 8-Feb-13 10:03am
   
Yes I'd consider ASIO reliable enough. Boost stuff is generally very good and the longer it's been around the better tested it is. They have a large user base so stuff gets a hammering and bugs get reported and pretty soon sorted out. The only problem with ASIO like much of Boost: it's code written by geniuses so you sometimes have to be a genius to work out how to use it. I would definitely do the test app to prove the concept. Hope it all goes well.
Rate this: bad
good
Please Sign up or sign in to vote.

Solution 1

On a 32 bit system, an int is 32 bits and any 'size' value will be determined by this as an upper limit (actually an unsigned int, not a signed int) but you will run into other limits far sooner than that. The 32 bit int (unsigned) also defines address space, and you don't have all of that available to you to store your data if the data needs to be in memory.
 
Since you are talking about streams, I assume that you do not want to put all of the data in memory at the same time.
 
The simplest way to count bytes in the stream is to use a 64 bit int. This should give you many years of countability (did I just invent a new word?).
  Permalink  
Comments
WaZoX at 7-Feb-13 14:57pm
   
I'm not sure you understood my question correctly? The TCP protocol itself does contain an integer in order to count the number of bits sent, from what I understand that limits the underlying TCP stream itself from sending more data than 4GB. So I wonder if the implementation of ASIO has solved this issue behind the scenes or if you should keep track of it yourself.
 
The data isn't stored in memory for very long, so not being able to address it in memory isn't a problem. The data is also "generated" in small chunks so that's neither a problem for the sender. I should have been more clear about that, thanks for pointing that out and for your answer.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

  Print Answers RSS
0 Sergey Alexandrovich Kryukov 6,725
1 OriginalGriff 6,696
2 CPallini 5,315
3 George Jonsson 3,589
4 Gihan Liyanage 2,650


Advertise | Privacy | Mobile
Web02 | 2.8.140921.1 | Last Updated 7 Feb 2013
Copyright © CodeProject, 1999-2014
All Rights Reserved. Terms of Service
Layout: fixed | fluid

CodeProject, 503-250 Ferrand Drive Toronto Ontario, M3C 3G8 Canada +1 416-849-8900 x 100