Click here to Skip to main content
15,867,308 members
Please Sign up or sign in to vote.
0.00/5 (No votes)
I am working on an application that uses Silverlight. The issue is based around the update process.

The update uses delegates (via InvokeAsync) which place calls to a web service.

It will call BeginInvoke X number of times and THEN when EndInvoke is called we find out if they failed or not.

Each one of these requests is a seperate HTTP request (practically a denial of service attack).

Say we send in 2000 updates, only about 500-600 will actually process and the rest wont.

I ran a wireshark capture and it appears that all the entries are being sent to the server and are acknowledged by the server.

My initial thought was to attempt to make this application run synchronously but that isn't possible for Silverlight without some kind of framework (at least that's what I've come to conclude) and there is a lot of politics behind that where I'm working. A coworker said it may not be necessary to do all that work if it may just be some IIS configurations that need to be changed.

I've tried changing the queue length for the server and the application pool, I've increased the number of threads per processor, checked to see if the CPU usage was limited (NoLimit vs Kill). I've been looking at this for a couple weeks and I'm completely lost. Any help would be greatly appretiated. Let me know if I've missed anything. Thank you!

Update: I should add that the server does not see any errors. The DBA's are as frustrated as I am because of the lack of error information. The application in debug mode returns CommunicationException after a random number of updates.

System.ServiceModel.CommunicationException was unhandled by user code
  Message=The remote server returned an error: NotFound.
  StackTrace:
       at System.ServiceModel.AsyncResult.End[TAsyncResult](IAsyncResult result)
       at System.ServiceModel.Channels.ServiceChannel.EndCall(String action, Object[] outs, IAsyncResult result)
       at System.ServiceModel.ClientBase`1.ChannelBase`1.EndInvoke(String methodName, Object[] args, IAsyncResult result)
       at WaterFrontApplication.WorkOrderServiceWrapper.WorkOrderServiceClient.WorkOrderServiceClientChannel.EndAssignCrew(String& fault, IAsyncResult result)
       at WaterFrontApplication.WorkOrderServiceWrapper.WorkOrderServiceClient.WaterFrontApplication.WorkOrderServiceWrapper.WorkOrderService.EndAssignCrew(String& fault, IAsyncResult result)
       at WaterFrontApplication.WorkOrderServiceWrapper.WorkOrderServiceClient.OnEndAssignCrew(IAsyncResult result)
       at System.ServiceModel.ClientBase`1.OnAsyncCallCompleted(IAsyncResult result)
  InnerException: System.Net.WebException
       Message=The remote server returned an error: NotFound.
       StackTrace:
            at System.Net.Browser.AsyncHelper.BeginOnUI(SendOrPostCallback beginMethod, Object state)
            at System.Net.Browser.BrowserHttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
            at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelAsyncRequest.CompleteGetResponse(IAsyncResult result)
       InnerException: System.Net.WebException
            Message=The remote server returned an error: NotFound.
            StackTrace:
                 at System.Net.Browser.BrowserHttpWebRequest.InternalEndGetResponse(IAsyncResult asyncResult)
                 at System.Net.Browser.BrowserHttpWebRequest.<>c__DisplayClass5.<EndGetResponse>b__4(Object sendState)
                 at System.Net.Browser.AsyncHelper.<>c__DisplayClass4.<BeginOnUI>b__1(Object sendState)
            InnerException:
Posted
Updated 19-Jul-13 5:54am
v3
Comments
Jason Gleim 19-Jul-13 14:22pm    
Are you specifically calling the async invoke passing in the web service proxy methods? If you generated the proxies from WSDLs, the proxy objects already have the async harnesses in place... you don't need to further async the call. (There is no such thing as a sync web service call in SL.) That's just an extra layer of indirection for no performance gain.

I've seen this before when the server gets overloaded with simultaneous calls. Depending on what you are working against, it may just silently drop the service request with an unhelpful error code. One thing to do is to use Fiddler instead of Wireshark. Fiddler will intercept the calls to and from the service rather than everything on the wire like Wireshark does.

If you make a Fiddler capture and need help looking at it, PM me and I'll send you my e-mail address.
MikeTheDebugger 19-Jul-13 15:10pm    
I believe I understand what you are saying... I am just calling one method that is in the proxy class and passing in a task and an ID. So, there is no further logic that is being created on my part. This is correct? Sounds awesome, I'll send you a PM, working on installing this program right now.
Jason Gleim 19-Jul-13 15:28pm    
I think we are talking the same thing. I assume you are calling SomeWebServiceClient.SomeMethod(param1, param2, callback); or something similar. In that case then yes, you are using the async that is already built into the proxy objects.

Some people read that all web services in SL have to be async and think they need to wrap the method calls in async code. I was afraid you had gone down that path.

I think you'll find Fiddler an amazing tool for debugging those calls. I don't leave home without it! :-)
MikeTheDebugger 19-Jul-13 18:29pm    
Yeah we're definitely on the same page :) I have a question about Fiddler: it seems to only capture about 440ish packets during my BeginInvoke (it starts failing after about 500-600). I might get some other activity captured but I no longer get anything from the application. The rest of the BeginInvokes do not show up and nothing from EndInvoke. I can't seem to find any limits on it... seems really odd. Otherwise, it looks really clean compared to Wireshark, I'll definitely be using Fiddler again.
Jason Gleim 19-Jul-13 20:03pm    
That makes me think there is a problem in the app. I've run Fiddler for hours capturing 1000s of sessions with no issues. And the fact you see other traffic tells me Fiddler isn't likely losing track of what is going on.

Two quick questions; Are all of these calls simultaneous? Meaning you see 400 to 500 OPEN sessions in Fiddler. Second, are you only setting the callback handler on your web method once? I suspect you may be hitting a threading limit or the ability of the web client to track active sessions in the first case. If you are setting the event handler for each call (without detaching it) you will end up with a super, super long eventing chain. Which, by the way, I've found in practice crashes somewhere around 500 entries. The webclient is static so if you call the proxy method and pass an event delegate in each call, you are loading up the eventing chain.

So it looks like the issue may be somewhere in the flood of async calls going to the server but it is actually breaking on the client side. This makes me curious and I may write up a test program just to dig into this deeper and maybe write an article about it. There is something going on there but I'm not exactly sure the specific issue.

I think there are two approaches to solving this but they are both, admittedly, workarounds. I think this is a framework issue and so solving it directly is out of the question in my opinion.

The first possible solution would be to throttle the calls limiting the number of simultaneous requests you send out. This could get pretty complicated. An implementation of this might be to setup a queue and when the tasks need processed, put each task on the queue in the foreach loop. Then, kick off a background worker thread which checks the queue for the presence of tasks. If tasks are present, the thread would pop x number of tasks off the queue and then execute the service method AWAITing (new async methods in the framework) the results (or kicking the methods off on worker threads themselves and joining the threads). The result is that you may execute 100 requests and wait for all 100 to complete. Then, check the queue count and start over. You could actually start the background worker with the app and let it run all the time. It is either sleeping or processing queued requests.

The drawback to this method is the complexity of building a background worker to throttle the calls out to the web service then marshaling the responses back to the UI. You would probably need some sort of static class to represent your task status or results that had thread-safe property accessors. Not impossible but fairly complicated and easy to miss something that would result in a cross-thread exception... every multithreading developer's favorite exception. :-P

Alternately, if you can change the web service that might be the best way to go. You could write an alternate version of the web service (polymorphism to the rescue!) which accepts an array of tasks. Then the client code becomes a single call passing in your list of tasks. The heavy lifting for iterating the tasks takes place on the server such that the whole thing is reduced to a single call and response.

This would be a more scalable solution as well I think. You could potentially run into the size limit of a POST message if you had lots and lots and lots of tasks to update at once. But on the client side you could set a maximum number of array elements, detect that in your source list, and make multiple-calls if you needed to. A scheme like that could potentially expand to 500 simultaneous calls times 5000+ array elements before you would have to look at another approach.

Unfortunately, neither of these are easy solutions but I think your app, as it stands now, has scaled into a framework limitation so you're going to have to backtrack out of this part of the maze and look for another route to the goal.
 
Share this answer
 
I took the event handler out and I'm sure I've tried that before. It's still not working right now. I got 700+ entries in Fiddler this time but, due to randomness of success, this could be less if I run this later on in the day. I've posted the original code and what I've done to it.

Contractor's code:
C#
if (tasksToSend.Count > 0)
{
    this.stoneTasks = tasksToSend.ToList();

    // Update Task
    foreach (AssignTaskCriteria task in tasksToSend)
    {
        this.MinimumNumberOfRequests = 0;
        this.MaximumNumberOfRequests = 0;
        this.NumberOfRequestsReceived = 0;
        this.IsIndeterminate = false;
        this.BusyContent = "Updating the task \n" + "Please wait...";
        this.IsBusy = true;
        var proxy = new WorkOrderServiceClient();
        proxy.AssignCrewCompleted += this.ProxyUpdateTaskCompleted;
        proxy.AssignCrewAsync(task, this.currentRequestUserStateID);
        this.MaximumNumberOfRequests++;
    }
}


My modifications:
if (tasksToSend.Count > 0)
{
    this.stoneTasks = tasksToSend.ToList();
    this.MinimumNumberOfRequests = 0;
    this.MaximumNumberOfRequests = 0;
    this.NumberOfRequestsReceived = 0;
    WorkOrderServiceClient proxy;
    proxy = new WorkOrderServiceClient();
    proxy.AssignCrewCompleted += this.ProxyUpdateTaskCompleted;
    // Update Task
    foreach (AssignTaskCriteria task in tasksToSend)
    {
        this.IsIndeterminate = false;
        this.BusyContent = "Updating the task \n" + "Please wait...";
        this.IsBusy = true;
        proxy.AssignCrewAsync(task, this.currentRequestUserStateID);
        this.MaximumNumberOfRequests++;
    }
}



Contractor's complete method... it only used to check the success of the first update because of that max = 0 at the beginning of the loop. I rewrote this whole method.
C#
public void ProxyUpdateTaskCompleted(object sender, AssignCrewCompletedEventArgs e)
        {
            if (((Guid)e.UserState).CompareTo(this.currentRequestUserStateID) == 0)
            {
                this.NumberOfRequestsReceived++;

                if (this.NumberOfRequestsReceived == this.MaximumNumberOfRequests)
                {
                    if (e.Error == null)
                    {
                        if (e.fault == null)
                        {
                            if (e.Result != null && e.Result.resultField.CompareTo("SUCCESS") == 0)
                            {
                                this.IsBusy = false;
                                this.DisplayAssignTaskMessage();
                                this.UpdateViewTaskDataGrid("AssignTask");
                                this.UpdateTaskOnMap();
                                this.CloseConfigAssignTask();
                            }
                            else
                            {
                                this.IsBusy = false;
                                this.ShowErrorMessage("Error updating the Task \n");
                            }
                        }
                        else
                        {
                            this.IsBusy = false;
                            this.ShowErrorMessage(string.Format("Error updating the Task: {0}", e.fault));
                        }
                    }
                    else
                    {
                        this.IsBusy = false;
                        this.ShowErrorMessage(string.Format("Error updating the Task: {0}", e.Error.Message));
                    }
                }
            }
        }
 
Share this answer
 
Comments
Jason Gleim 22-Jul-13 11:31am    
This is very helpful... let me digest the code a bit and I'll get back to you. In the meantime, do you have access to the code on the web service side? Meaning, could you redo the service endpoint to accept an array of AssignTaskCriteria? It would be a lot more efficient then sending them one at a time.
MikeTheDebugger 22-Jul-13 11:57am    
I don't but I might be able to change that. I feel like that would probably solve everything. It was brought up in a meeting but someone shut that down. "They wouldn't go for that" or something. I don't know if he was refering to the Web Service Admin guys or what (???). When this was originally written, it wasn't meant to update a substantial number of tasks but things have changed.
Jason Gleim 22-Jul-13 13:32pm    
I think you could make a few changes to the code you posted that would help but I'm beginning to think there is a practical limit there in the number of simultaneous sessions you can get away with. I know there is an IIS setting which adjusts the max number of concurrent sessions but if that was the issue, you should get 403 errors... not 404s... and we've seen with Fiddler that its not breaking down at the server anyway.

Your improved version of the calling loop is much better. But I think you can go ahead and move the UI stuff out too... no reason to set the indeterminate flag and IsBusy every time you iterate the loop. Better to set the UI stuff before you go into the loop since you already know you have at least one task to process. Avoids the overhead of unneeded thread marshaling.

I see though, in the callback method, a line which checks if the receive count matches the sent count. That is probably bad logic because it would short-circuit processing on a big number of calls. If the service takes any time at all, you could have 200 sent and the callback would not process the inside of that if statement until the 200th response came back. There may be some reason for doing this... without the complete context it is hard to know... but it seems like your contractor was trying to do some session matching or something. You could probably remove that thinking about how the event handler should respond to each callback for each individual session.

But, neither of those thing will fix your problem I think. I don't see anything fundamentally wrong with the revised code that could cause issues 500+ calls out. So we have to look at the concurrency of the calls. I'm going to post my suggestions as an answer... stand by.
MikeTheDebugger 23-Jul-13 12:07pm    
I posted something that I am working on as a solution.
I don't know if this is really "proper" but your explanation of events helped me come up with this "bandaid." It seems to work! I just successfully updated 2317 records. I watched them all go through Fiddler and then queried the database to confirm. If you have any ideas for improving this let me know but the basic idea is laid out. I plan on putting in some more error handling for one thing.

Instead of looping through the tasks in a foreach loop, this is called once in the method that executes on the assign tasks/continue button.

It is then called again in the completed method when the condition is met that we are on the last one sent.

0-99
100-199
200-299 ...

XML
/// <summary>
/// Send only MAX 100 items from the list at one time. Will be recalled until update is complete.
/// </summary>
public void sendFragmentOfList()
{
    AssignTaskCriteria taskToSend;
    int current = this.newStart;

    WorkOrderServiceClient proxy;
    proxy = new WorkOrderServiceClient();
    proxy.AssignCrewCompleted += new EventHandler<AssignCrewCompletedEventArgs>(ProxyUpdateTaskCompleted);
    this.newStart += 100;

    while (current < this.newStart)
    {
        if (current < MaximumNumberOfRequests)
        {
            taskToSend = this.stoneTasks.ElementAt(current);
            proxy.AssignCrewAsync(taskToSend, this.currentRequestUserStateID);
        }
        current++;
    }

}


Snippet from the completed method. If we are on the LAST task then a different if statement is executed before that notifies the user of results.

C#
if (this.NumberOfRequestsReceived == this.newStart - 1)
    sendFragmentOfList();
 
Share this answer
 
Comments
Jason Gleim 23-Jul-13 16:29pm    
This will absolutely work as well! I've used a similar technique before chunking large uploads to a web service. There is nothing wrong with chaining a new call from the completed event.

This should scale well for you if it ever needed to. Assuming there is a break between the call and the callback (which there would have to be), the UI thread will gain back control which would unwind the initial call stack. This means you wouldn't have to worry about "Inception" style recursive callback stacking like you had before when it was all in a single foreach loop.

Glad you hear you found a solution!
MikeTheDebugger 23-Jul-13 16:38pm    
Thank you very much for all the help! I couldn't have done this without hearing your explaination of just what's going on with the code & events and your suggestions! You said you may write something about this and/or do some further testing, I would be interested in hearing about whatever you come up with. Let me know! Cheers again!

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900