In config with standard http or wsHttp bindings, you would usually do this:
<binding name="myBinding">
<readerquotas maxarraylength="2147483647"></readerquotas>
</binding>
In code with a custom binding, you need to add a MessageEncoding element. You've done the transport layer, you need to go up a step, like so (I'm assuming here you'll use binary message encoding which is good for file transfer):
System.ServiceModel.Channels.CustomBinding binding = new System.ServiceModel.Channels.CustomBinding();
binding.Elements.Add(new System.ServiceModel.Channels.TextMessageEncodingBindingElement(System.ServiceModel.Channels.MessageVersion.Soap11, System.Text.Encoding.UTF8));
System.ServiceModel.Channels.HttpTransportBindingElement http = new System.ServiceModel.Channels.HttpTransportBindingElement();
http.MaxBufferPoolSize = 300000;
http.MaxBufferSize = 300000;
http.MaxReceivedMessageSize = 300000;
http.TransferMode = System.ServiceModel.TransferMode.Buffered;
binding.Elements.Add(http);
var messageElement = new System.ServiceModel.Channels.BinaryMessageEncodingBindingElement();
messageElement.ReaderQuotas.MaxArrayLength = Int32.Max;
binding.Elements.Add(messageElement);
return binding;
The error message stating
"This quota may be increased by changing the MaxArrayLength property on the XmlDictionaryReaderQuotas object used when creating the XML reader" is because underneath the framework is instantiating an XmlReader with the quotas specified in the binding element. Once you add your message encoding element, the XmlReader it then creates will have a larger MaxArrayLength :)
Hope that helps! :)