OutOfMemoryException while bulk data processing with WCF RIA & WF4

OutOfMemoryException while bulk data processing with WCF RIA & WF4

I have an existing Silverlight 5 application. I'm adding a page to it to allow users to process mass updates to data in a 3rd party database system. The application currently uses WCF RIA services to communicate to the 3rd party system via SOAP. The functionality of the update is contained in a Workflow 4 application I created and is referenced as an assembly on the server-side of the SL application. Lastly, the application is hosted right now in my local instance of IIS 7.5 running on Windows 7; I'm also debugging with IIS, not the VS dev server.

At the basic level, the application functions as follows:

• Select text file
• Click "Start" button
• Event handler creates an instance of a user-defined Type that keeps track of the batch
• Event handler creates a new BackgroundWorker instance and wires up handlers for the DoWork, ProgressChanged, and RunWorkerCompleted events
• Event handler calls RunWorkerAsync()

Here's the shortened code for the DoWork event handler, since that's where the majority of the work is done.

private void BwOnDoWork(object sender, DoWorkEventArgs doWorkEventArgs, BatchContainerControl batchProcess) {    var worker = sender as BackgroundWorker;    // Iterate through each record of data file and call the 'UpdateAddress' function    // of the AddressDomainService which, in turn, executes the Workflow    foreach (var item in batchProcess.FileData)    {       // Check if operation has been cancelled       if (worker.CancellationPending)       {           doWorkEventArgs.Cancel = true;           break;       }      . . .        // Invoke THINKComm.CustomerAddressUpdate Workflow via AddressContext       var invokeOp = _addressDomainContext.UpdateAddress(activityData);       // 'activityData' is an instance of Dictionary<string, string>        invokeOp.Completed += (o, args) => InvokeOpOnCompleted(o, args, batchProcess);    } } 

The handlers for the ProgressChanged and RunWorkerCompleted events, as well as the Completed event of the InvokeOperation instance all, for the most part, update a part of the UI. If you think posting any of that code would be helpful, I'd be happy to update the post.

Speaking of UI, the parts that are updated by the event handlers are two ProgressBar controls - one that tracks the records as they're read from the file and a second one that tracks the records as the update has taken place on the 3rd party database.

Getting to the actual problem...

I've processed files of 10, 100, and 1,000 records with no problem. I then attempted to process a complete file containing ~15,000 records (or 1,907KB of data). The process starts and I can see in the debugger output that the Workflow is being executed. About a quarter of the way through or so, I get an OutOfMemoryException. Here's the stack trace:

at System.ServiceModel.DomainServices.Client.WebDomainClient`1.BeginInvokeCore(InvokeArgs invokeArgs, AsyncCallback callback, Object userState) at System.ServiceModel.DomainServices.Client.DomainClient.BeginInvoke(InvokeArgs invokeArgs, AsyncCallback callback, Object userState) at System.ServiceModel.DomainServices.Client.DomainContext.InvokeOperation(String operationName, Type returnType, IDictionary`2 parameters, Boolean hasSideEffects, Action`1 callback, Object userState) at THINKImportSystem.Web.Address.AddressDomainContext.UpdateAddress(Dictionary`2 activityData) at THINKImportSystem.BatchProcessPage.BwOnDoWork(Object sender, DoWorkEventArgs doWorkEventArgs, BatchContainerControl batchProcess) at THINKImportSystem.BatchProcessPage.<>c__DisplayClass10.<StartButtonClick>b__6(Object s, DoWorkEventArgs args) at System.ComponentModel.BackgroundWorker.OnDoWork(DoWorkEventArgs e) at System.ComponentModel.BackgroundWorker.OnRun(Object argument) 

Then, the JIT debugger pops up with an error of Unhandled Error in Silverlight Application Code:4004 with a message of System.ServiceModel.DomainServices.Client.DomainOperationException: Invoke operation 'UpdateAddress' failed. Error HRESULT E_FAIL has been returned from a call to a COM component.

I should mention that, sometimes, I get the JIT debugger first. I see in the Debug output that threads are still exiting, and then about 10 or 20 seconds later, the VS debugger pops up with the out of memory exception.

My best guess is that, objects somewhere (maybe related to the DomainService?) aren't being released and therefore, memory usage is building. From what I understand, IIS places restrictions on the amount of memory an application can use, but I can't tell if that's the case here or not.

I was thinking that, each time a record in the file is processed, the objects related to it's processing would be released and therefore overall memory usage would be pretty low. But obviously I'm not understanding how everything is being executed!

I was also wondering if using the TPL as opposed to BackgroundWorker would make a difference?

Answers & Comments...

---