|
The best possible solution, assuming you catch batch the data, is to use the ADO.NET stuff that's designed to talk to databases. Flat files on the machine are a terrible idea.
If it was me, I'd write the code to create a similar dataset, and stuff it with 20 million dummy records, to see what it does. However, if it was me, I'd never grab that many records anyhow, I'd make all operations on the data happen in the database, and my app would never retrieve more data than it needed to display on one page.
Christian Graus - Microsoft MVP - C++
|
|
|
|
|
HI Christian Graus,
Thanks for the info,
Here I had to get all that data. I am not displaying any data in my application(just for testing i am using datagrids)..I just have to process it and put the resultent data in DB2.
Thanks and Regards,
Subbu
|
|
|
|
|
Why do you need to process it all at once ? What's the process, that you can't do it within the database ?
Christian Graus - Microsoft MVP - C++
|
|
|
|
|
dvsr wrote: Please Suggest the best possible solution.
15 million records might be a lot of records to process at once, its hard to tell without seeing the overall picture(there might be other compelling solutions). Even before looking for specific tech to use, see if you can do the processing in piecemeal and save the result and use it to work with the next batch. Though I think the previous suggestion of doing the processing on the DB side might be an issue if other apps are using it at the same time.
Quran
Translation
Intro
Discover
|
|
|
|
|
It's best to leave the processing in the database itself. Retrieving/Sending 30M records (you have to put them back ater the processing, correct?) will (usually) needlessly beat the crap out of the network your SQL server is attached to.
Part of the job of a developer is to evaluate your processes and code and consider their impact on the resources around them. What is this servers SQL load? What about it's NIC? What is the impact of adding 30M transactions to those loads? (I mean either from your processing application or from doing it inside the SQL database!) Will your retrieval, processing, and upload spike the use of these resources so bad that normal traffic is negatively impacted? How long will this processing take using each method? ...?, ...?, ...?, ...?
The .NET Framework dataset object can handle as much data as you can fit in memory. Millions of records is not out of the question. The real question is if it's practical. Only you can say.
RageInTheMachine9532
"...a pungent, ghastly, stinky piece of cheese!" -- The Roaming Gnome
|
|
|
|
|
HI Dave,
Thanks for ur suggesition.
My Applications is something related to Airlines business. We expect atleast 15M records per day .... We r doing some statistical analysis on that data and the resultent set will be very small which we r going to send back to DB2.
BTW ..we r using DB2. ANyway I can rise the issues and discuss this suggesitions in my team meeting.
Thanks alot.
Subbu
|
|
|
|
|
How much data is it? If each field contain an average of 10 bytes, it comes to about 16 GB. Plus some overhead. I doubt that you have a machine with that amount of RAM...
Do you really need all the data at once in memory to process it?
If you need all the data at once, you probably have to copy the data into a temporary table, process it there, then read the result from it.
---
b { font-weight: normal; }
|
|
|
|
|
Hi All
I want write some method in c# with default parameters.
I try to do it like i did in C++ but the compiler does not let me.
How can i do it ?
Thanks .
|
|
|
|
|
Nope, VB.NET has this, and C# doesn't. Funny enough, I raised this at the MVP summit to the guys who design the language, and didn't get a flood of support.
The only way to do this is to write two methods, one that takes the extra parameter and one that does not. Make sure they both call the same underlying function.
Christian Graus - Microsoft MVP - C++
|
|
|
|
|
Do it with overload method is not smart way.
The default parameters is the best way to do it.
|
|
|
|
|
yanshof wrote: The default parameters is the best way to do it.
I agree - if C# supported default parameters, it would indeed be the best way to do it.
Christian Graus - Microsoft MVP - C++
|
|
|
|
|
IS there is some way OR some mail that we [the develop user] can write [by mail] to microsoft ?
|
|
|
|
|
yanshof wrote: IS there is some way
Yes. I told you there wasn't any way because I wanted to test you.
Actually, there is no way, just like I said 
yanshof wrote: some mail that we [the develop user] can write [by mail] to microsoft ?
I'm sure there are ways to write to Microsoft, and I'm sure they would give it some consideration, but certainly I've put my vote in, in person, as an MVP, I'm not sure that an email on the topic would help more than that, they certainly didn't respond strongly to the suggestion, and nor did any of the C# MVPs present, which left me feeling that there's probably not many who moved from C++ and understand how useful this can be. Certainly, I said that I think C# errs too often on holding the developers hand ( and they may feel that optional parameters can cause coding errors ), and they said that will definately continue to be the case.
Christian Graus - Microsoft MVP - C++
|
|
|
|
|
There is no difference between the two, for example
public foo(bool someVariable)<br />
{<br />
foo(someVariable, false);<br />
}<br />
public foo(bool someVariable, bool anotherVariable)<br />
{<br />
}<br />
Optional parameters are not CLS compliant and are therefore not a great idea
|
|
|
|
|
Unfortunatley in .NET Just C# does not support Default parameters.
|
|
|
|
|
Hi,
I am trying to develop a web application, and I am using Third Party COM DLL. On running The application it works until the where it initiates class. I have used Tlbimp to create a wrapper too. Also it was suggested to add IWAM_account to the impersonate security policy for local settings. I have update the group policy using force, and also restarted an IIS, but still no luck with it.
Code looks like
<br />
using dllname;<br /> creating object
dllname.obj some = new dllname.obj(); <----- Giving me error here at this line Interop Com Exception catastrophic error.
Any help will be appreciated. Thanks,
|
|
|
|
|
This is a COM error, it means that your dll is running ito an error at initialisation, which I assume means you need to set something else up before you create it.
Christian Graus - Microsoft MVP - C++
|
|
|
|
|
Thanks for giving me your insight on this.
I am still not clear how can the same DLL is working with ASP using Server.creta Object. But when I want to use for ASP.net, and I am using the code behind as C#.net I am running to an issue. There got to be a way to solve this problem. However, if you can give me any technical information specifically I can talk to the developer who developed this COM I can talk to him to check. Thanks,
|
|
|
|
|
I have no idea why it's blowing up. If you can talk to the person who wrote it, perhaps they can help. One possibility is that I think the threading model of ASP.NET is fixed, perhaps the model of the component needs to change. However, I'm taking a stab in the dark. If you have access to the component's author, then definately being able to add some debugging code or otherwise work out where the exception is thrown is a lot more useful than trying to deal with a black box.
Christian Graus - Microsoft MVP - C++
|
|
|
|
|
I am trying to find out the version numbers of the components that appear in the Add/Remove programs list.
The data I'm after is stored in the registry BUT I don't seem to be able to access it via code but I can view it usingg Regedit.
Below is the code I'm using to try and access the data. As you can see, I've tried setting the access permission but I'm not entirely sure if my implementation is correct.
If I trim the registry path down to "Software\\Microsoft\\Windows" I get a key returned.
Any ideas???
Cheers,
Ben
<code>
/// <summary>
/// Create a registry key object to allow us to retrieve installed components version numbers.
/// </summary>
/// <returns>The registry key to use if no error occured, else null</returns>
private RegistryKey GetRegistryKey()
{
RegistryKey sysSettingsKey = null;
try
{
// Setup permission to ccess the key
RegistryPermission f = new RegistryPermission(PermissionState.Unrestricted);
f.AddPathList(RegistryPermissionAccess.Read,
"HKEY_LOCAL_MACHINE\\SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Uninstall");
// Try and open the key to see if it exists
sysSettingsKey = Registry.LocalMachine.OpenSubKey("Software\\Microsoft\\Windows\\Current Version\\Uninstall");
// Was the key found?
if( sysSettingsKey == null )
{
MessageBox.Show("Unable to access the uninstalll information", "Registry Error", MessageBoxButtons.OK, MessageBoxIcon.Exclamation);
}
}
catch(Exception ex)
{
MessageBox.Show(string.Format("Exception Detected: \n{0}",ex.ToString()),"Registry Error", MessageBoxButtons.OK, MessageBoxIcon.Exclamation);
}
return sysSettingsKey;
}
</code>
-- modified at 14:43 Thursday 20th October, 2005
|
|
|
|
|
You're gonna kick yourself.
Wild _Wildy wrote: sysSettingsKey = Registry.LocalMachine.OpenSubKey("Software\\Microsoft\\Windows\\Current Version\\Uninstall");
This works
Wild _Wildy wrote: sysSettingsKey = Registry.LocalMachine.OpenSubKey("Software\\Microsoft\\Windows\\Current Version\\Uninstall");
This needs the space between 'Current' and 'Version' removed, so it's the same as the line above, then it works.
Christian Graus - Microsoft MVP - C++
|
|
|
|
|
Thanks very much.
I didn't quite kick myself (that would just be silly) but I did drink a few beers and have a laugh about the smallest errors being the hardest to find.
I had got the path from Regedit using the "Copy Key Name" menu item so I don't feel like it was my typo.
Thanks again and Cheers. 
|
|
|
|
|
We have a web service that is growing large enough that we wish to seperate logic. Ideally, we would have multiple code files on the server, and the service itself would still appear to be the same to consumers of it.
If we were doing it in c++, we would simply split the implementation of our WebService class across several .cpp. Of course, we can't do that in C#.
Any ideas or pointers to docs?
|
|
|
|
|
Well, in VS2005/C#2.0 you have partial classes if thats what you mean (spreading the implimentation across files). But I don't see why you couldn't just have the webservice methods call methods in other classes.
Matt Newman
Even the very best tools in the hands of an idiot will produce something of little or no value. - Chris Meech on Idiots
|
|
|
|
|
Another part of it is wanting to group methods by thier function; e.g. multiple webservices in one. So from the WSDL, you'd have
Service1<br />
{ <br />
func1();<br />
func2();<br />
}<br />
<br />
Service2<br />
{<br />
func1();<br />
func2();<br />
}
Your information about 2005 helps though - that may end up being our solution.
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
