How to minimize UDP Packet Loss

Question

0.00/5 (No votes)

See more:

Hi,

Here is my problem:
I try to send a lot of data (big images) between 2 PC using UDP socket (UDP is the only choice for Multicast).

I use something like RUDP (Reliable-UDP), basically:
- the Server sends a lot of UDP packet with id (one per image) and packet index
- the Client receives the packets and reconstruct the image using the packet index
- if packets are missing, the client sends an UDP message to server to request the missing packets
- the Server sends the missing packet
this loop of actions is done until the server must send the next image.

When everything works fine, a few number of packets is missing and the network used only 5% of bandwidth.
But sometimes, almost all packets are missing and the network used more than 25% of bandwidth (limitation is due to timeout in my implementation, and this work like I want) and represent sending 5 times the same image!

I tried :
- deactivating anti-virus, firewall
- direct connection

- checking if Windows Filtering Platform (WFP) dropped some packet, but nothing appears in the log when firewall is off.

- If I look the resource monitor on both server and client, network use are identical.

I knows it should be possible because I can connect a device sending twice the data framerate I tried without packet loss.

How can I investigate where the packets are lost/dropped?

Posted 24-Apr-15 5:15am

Pascal-78

Updated 6-Jul-17 21:40pm

Add a Solution

Comments

Richard MacCutchan 24-Apr-15 13:10pm

Switch to TCP.

Pascal-78 24-Apr-15 13:21pm

And how do you do "Multicast" with TCP ?

Homero Rivera 24-Apr-15 13:26pm

Explain "multicast". What are the dynamics between clients and the server?

TCP can definitely "multicast".

How are the clients signaled that there is an image to receive? Or do the clients ask for the images every some time?

Pascal-78 24-Apr-15 13:36pm

Multicast (http://en.wikipedia.org/wiki/Multicast) is a way to send data (in my case) from 1 PC to many others but not all.
Unicast: 1 to 1
Multicast: 1 to many, many to many
Broadcast: 1 to all.

My current test is made only with 2 PC (1 sender, 1 receiver) but the main goal of my project will use more receivers.

The advantage of Multicast from TCP: it used only 1 send for all the clients where TCP needs 1 send per client. In terms of bandwidth : Multicast does not change the needed bandwidth if the number of clients increase, but TCP will have to use N times the bandwidth for N clients.

This application is similar to an ip-camera sending images to clients, the clients do not ask for image, clients are listening on an opened port for the incoming stream.

Homero Rivera 24-Apr-15 13:43pm

I understand what a multicast is in networking :)
What I'm wondering is how the clients know there is an image they should be getting? Or are the clients always open with a service to get those images whenever the server sends to them?

Wouldn't it be easier if the clients had an application with a timer, where every so many minutes they checked if the server has any new images? and then download them if there are any? And the download occurred via TCP?

That's what I mean by dynamics.

I think all your clients are being used as servers. Clever, but, evidently you are running into trouble.

If bandwith is an issue, then what should be the real server (the one providing images) could serve the requests one by one and not all at once, by putting come clients requests on hold until it it is their turn.

Pascal-78 24-Apr-15 14:39pm

The clients are polling their socket to check if new data are available.
When data are available, they retrieve them.
And yes, it looks like a server, but with your proposal, it quite similar:
- client send request to server using TCP
- client receive data from server using TCP
in my case, there is no request sent by client, and the client receives data from server using UDP.
You talk about minutes, I talk about millisecond. in fact, the server (the one providing the images) should send a new image every 22ms (approximately 45 img/s), each image is more than 2MB (and yes, that means more than 700Mb/s, but I use a 10Gb/s link between the PC.).
And the clients should do their job in realtime. I cannot delay one client...

Homero Rivera 24-Apr-15 14:54pm

Oh I see! You are making a live transmission application. Like TV over the internet or sharing your desktop or camera with multiple users huh?

Is that what this is about?

Pascal-78 24-Apr-15 15:03pm

It's quite the idea,
but for sharing desktop to multiple users, we have found hardware solutions,
our cameras already have their multicast mode, but if we want to insert some image processing, we have to send a new image from PC to others or if we want to simulate a camera...

Homero Rivera 24-Apr-15 15:07pm

Then definitely TCP is not the solution; if I'd knew that from the beginning!

Have you tried any of this tools to troubleshoot your network?
http://blog.pluralsight.com/network-troubleshooting-tools

Do you think any of them could work this one out for you?

Pascal-78 24-Apr-15 15:16pm

I check the link you gave, but since I made tests using direct-link, a lot of the tools are irrelevant. SpeedTest on a Local Area Network will be usefull but not over Internet...

Homero Rivera 24-Apr-15 15:08pm

One additional question... How do your clients connect to the streaming server? Is it via web? Is a DNS call made on every single request for the image?

One more thing Pascal... I'm really interested to help you with this problem to the best of my abilities, even if it takes us days.
This is the kind of challenges that brought me here, it helps me look at things that I didn't think of before.
It'll be a pleasure to go on and on with this problem.

Pascal-78 24-Apr-15 15:24pm

The clients connect themself to the stream by registering to a multicast group (ip from 224.0.0.0 to 239.255.255.255). They all use the same multicast address, and the server send images to this multicast address.
When I'm using only 1 client, the client do not register to a multicast group, and the server send the images to the client ip address directly.
I have a real TCP/IP server running to configure the image sender from a client or to retrieve the multicast group from a new client.

Any help will be appreciate. I'm working on it for weeks now. It will be a good article if we successfully solve this problem.

Homero Rivera 24-Apr-15 15:48pm

This article shows promise as a structured way to troubleshoot the network and packets ( https://documentation.meraki.com/zGeneral_Administration/Tools_and_Troubleshooting/Troubleshooting_packet_loss_between_devices ).

Maybe you have used pings and things like that, but I didn't know before you can increase the number of packets sent BY THE MILLIONS!
So, if you try 1000 packets and you didn't get any significant loss of packets, then you probably don't have a network issue.

Then the article gets more interesting, cause if you do get a huge loss of packets you can use tracert to isolate the point where the loss occurs.

No need for fancy tools, just old Command Prompt.

Have you tried all of this already?

Pascal-78 27-Apr-15 3:39am

I tried ping with -t option for continuous test, and no packet is lost with ping but in the same time, my program still losing packets.
And if you use ping with -n 1000, you will have to wait 1000 seconds to get the result. ping makes request every second.
I'm quite sure the problem is not on the network but in the IP stack of Windows, or the Windows Filtering Platform, or in my program itself...

I found this program: https://iperf.fr/
and its results are similar to my program, sometime 88% of datagram loss, sometime 0%...

2 solutions

Solution 4

I have a similar problem, but I can withdraw images with packets lost.
I solved the problem making following steps:
1. Increasing the socket buffer size as said Pascal-78
2. In the receiver part I made a separate thread that only receives packets
3. Maintain the link live by sending back an acknowledge command every time I receive a command
4. Using select() with a timeout between 10 and 500 microseconds to look for a packet before reading it
5. As my soultion must work also in linux I made following command to allow increase the socket size:

sudo sysctl -w net.core.rmem_max=myMaxImagesize*2

6. I wrote at console a message every time data arrives, but unfortunately windows provokes a great delay writting at console. I solved that by writting this code lines at the beginning of the main() to increase console buffer:

#ifndef __linux__   //Introduce this code at the beginning of main() to increase a lot the speed of cout in windows: 
	char buf[4000]; setvbuf(stdout, buf, _IOFBF, sizeof buf);
#endif

Posted 6-Jul-17 21:40pm

Javier Luis Lopez

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Pascal-78 · Accepted Answer · 2015-04-28T05:45:00

The solution was too simple:

increase the Socket Option ReCeiVe BUFfer (also named SO_RCVBUF).

by default, this buffer is 8kb, in my case it is the size of one UDP packet. That's mean, when my client reads the packet it must be done before the server sends a new packet or the new packet will be discarded by the IP stack.

I increased the buffer to a bigger value (twice the size of an image) using the setsockopt function:

C++

int rcvbufsize = myBigImageSize*2;
setsockopt(mySocket,SOL_SOCKET,SO_RCVBUF,(char*)&rcvbufsize,sizeof(rcvbufsize));

I increased the speed of my server (by waiting less between each packet) and no packet loss!

Now, I need to optimize the size of this receive buffer.