Re: Lorge volume ...Can .NET Handle ? - C# Discussion Boards

Lorge volume ...Can .NET Handle ?

20-Oct-05 12:05

I have been offered a job writing an image viewer for a web app. Now, I'm not sure yet, but I suspect they will want to be able to click a link in their web app, and have an image viewer come up on the client. Can I do this in C# ( set it up so that when I click a link, my app gets installed if need be, and then run using the image from the web, even though it's not associated with this image type ) ?

Christian Graus - Microsoft MVP - C++

dvsr20-Oct-05 11:47

dvsr

20-Oct-05 11:47

Hi,
I am writting an application in C#.NET.This application will run daily once.
The application has to fetch the data from DB2. The volume may be 15 million records(each record will be having 120 fields). I will be processing that data and the resultent data will be sent to DB2(diff table). Will ".Net"(dataset) can handle that much volume of data?

Suppose the data is in some flat file(in some machine) is file handling functions are best way to use.

Please Suggest the best possible solution.

-- modified at 18:02 Thursday 20th October, 2005

Re: on .net

Christian Graus20-Oct-05 12:02

20-Oct-05 12:02

The best possible solution, assuming you catch batch the data, is to use the ADO.NET stuff that's designed to talk to databases. Flat files on the machine are a terrible idea.

If it was me, I'd write the code to create a similar dataset, and stuff it with 20 million dummy records, to see what it does. However, if it was me, I'd never grab that many records anyhow, I'd make all operations on the data happen in the database, and my app would never retrieve more data than it needed to display on one page.

Christian Graus - Microsoft MVP - C++

Re: on .net

dvsr20-Oct-05 12:10

dvsr

20-Oct-05 12:10

HI Christian Graus,

Thanks for the info,

Here I had to get all that data. I am not displaying any data in my application(just for testing i am using datagrids)..I just have to process it and put the resultent data in DB2.

Thanks and Regards,
Subbu

Re: on .net

Christian Graus20-Oct-05 15:14

20-Oct-05 15:14

Why do you need to process it all at once ? What's the process, that you can't do it within the database ?

Christian Graus - Microsoft MVP - C++

A.A.20-Oct-05 15:07

A.A.

20-Oct-05 15:07

dvsr wrote:
Please Suggest the best possible solution.

15 million records might be a lot of records to process at once, its hard to tell without seeing the overall picture(there might be other compelling solutions). Even before looking for specific tech to use, see if you can do the processing in piecemeal and save the result and use it to work with the next batch. Though I think the previous suggestion of doing the processing on the DB side might be an issue if other apps are using it at the same time.

Quran
Translation
Intro
Discover

Dave Kreskowiak20-Oct-05 22:43

Dave Kreskowiak

20-Oct-05 22:43

It's best to leave the processing in the database itself. Retrieving/Sending 30M records (you have to put them back ater the processing, correct?) will (usually) needlessly beat the crap out of the network your SQL server is attached to.

Part of the job of a developer is to evaluate your processes and code and consider their impact on the resources around them. What is this servers SQL load? What about it's NIC? What is the impact of adding 30M transactions to those loads? (I mean either from your processing application or from doing it inside the SQL database!) Will your retrieval, processing, and upload spike the use of these resources so bad that normal traffic is negatively impacted? How long will this processing take using each method? ...?, ...?, ...?, ...?

The .NET Framework dataset object can handle as much data as you can fit in memory. Millions of records is not out of the question. The real question is if it's practical. Only you can say.

RageInTheMachine9532
"...a pungent, ghastly, stinky piece of cheese!" -- The Roaming Gnome

dvsr21-Oct-05 3:34

dvsr

21-Oct-05 3:34

HI Dave,
Thanks for ur suggesition.
My Applications is something related to Airlines business. We expect atleast 15M records per day .... We r doing some statistical analysis on that data and the resultent set will be very small which we r going to send back to DB2.

BTW ..we r using DB2. ANyway I can rise the issues and discuss this suggesitions in my team meeting.

Thanks alot.
Subbu

Default parameters in C# method

Guffa21-Oct-05 3:16

Guffa

21-Oct-05 3:16

How much data is it? If each field contain an average of 10 bytes, it comes to about 16 GB. Plus some overhead. I doubt that you have a machine with that amount of RAM...

Do you really need all the data at once in memory to process it?

If you need all the data at once, you probably have to copy the data into a temporary table, process it there, then read the result from it.

---
b { font-weight: normal; }

Yanshof20-Oct-05 11:04

Yanshof

20-Oct-05 11:04

Hi All

I want write some method in c# with default parameters.
I try to do it like i did in C++ but the compiler does not let me.

How can i do it ?

Thanks .

Christian Graus20-Oct-05 12:03

20-Oct-05 12:03

Nope, VB.NET has this, and C# doesn't. Funny enough, I raised this at the MVP summit to the guys who design the language, and didn't get a flood of support.

The only way to do this is to write two methods, one that takes the extra parameter and one that does not. Make sure they both call the same underlying function.

Christian Graus - Microsoft MVP - C++

Yanshof20-Oct-05 12:07

Yanshof

20-Oct-05 12:07

Do it with overload method is not smart way.
The default parameters is the best way to do it.

Christian Graus20-Oct-05 12:10

20-Oct-05 12:10

yanshof wrote:
The default parameters is the best way to do it.

I agree - if C# supported default parameters, it would indeed be the best way to do it.

Christian Graus - Microsoft MVP - C++

Yanshof20-Oct-05 12:36

Yanshof

20-Oct-05 12:36

IS there is some way OR some mail that we [the develop user] can write [by mail] to microsoft ?

Christian Graus20-Oct-05 12:49

20-Oct-05 12:49

yanshof wrote:
IS there is some way

Yes. I told you there wasn't any way because I wanted to test you.

Actually, there is no way, just like I said Smile | :)

yanshof wrote:
some mail that we [the develop user] can write [by mail] to microsoft ?

I'm sure there are ways to write to Microsoft, and I'm sure they would give it some consideration, but certainly I've put my vote in, in person, as an MVP, I'm not sure that an email on the topic would help more than that, they certainly didn't respond strongly to the suggestion, and nor did any of the C# MVPs present, which left me feeling that there's probably not many who moved from C++ and understand how useful this can be. Certainly, I said that I think C# errs too often on holding the developers hand ( and they may feel that optional parameters can cause coding errors ), and they said that will definately continue to be the case.

Christian Graus - Microsoft MVP - C++

PaulPrice23-Jul-07 9:50

PaulPrice

23-Jul-07 9:50

There is no difference between the two, for example

public foo(bool someVariable)<br />
{<br />
//Call foo with default variable<br />
foo(someVariable, false);<br />
}<br />
public foo(bool someVariable, bool anotherVariable)<br />
{<br />
//do something<br />
}<br />

Optional parameters are not CLS compliant and are therefore not a great idea

Ali Beirami21-Oct-05 3:00

Ali Beirami

21-Oct-05 3:00

Unfortunatley in .NET Just C# does not support Default parameters.

Catastrophic Error

NET_GEEK20-Oct-05 11:01

NET_GEEK

20-Oct-05 11:01

Hi,

I am trying to develop a web application, and I am using Third Party COM DLL. On running The application it works until the where it initiates class. I have used Tlbimp to create a wrapper too. Also it was suggested to add IWAM_account to the impersonate security policy for local settings. I have update the group policy using force, and also restarted an IIS, but still no luck with it.

Code looks like

<br />
using dllname;<br />

creating object

dllname.obj some = new dllname.obj(); <----- Giving me error here at this line Interop Com Exception catastrophic error.

Any help will be appreciated. Thanks,
Smile | :)

Re: Catastrophic Error

Christian Graus20-Oct-05 12:03

20-Oct-05 12:03

This is a COM error, it means that your dll is running ito an error at initialisation, which I assume means you need to set something else up before you create it.

Christian Graus - Microsoft MVP - C++

Re: Catastrophic Error

NET_GEEK20-Oct-05 17:59

NET_GEEK

20-Oct-05 17:59

Thanks for giving me your insight on this.
I am still not clear how can the same DLL is working with ASP using Server.creta Object. But when I want to use for ASP.net, and I am using the code behind as C#.net I am running to an issue. There got to be a way to solve this problem. However, if you can give me any technical information specifically I can talk to the developer who developed this COM I can talk to him to check. Thanks,

Laugh | :laugh:

Re: Catastrophic Error

Christian Graus23-Oct-05 12:13

Re: Reading Registry Data

23-Oct-05 12:13

I have no idea why it's blowing up. If you can talk to the person who wrote it, perhaps they can help. One possibility is that I think the threading model of ASP.NET is fixed, perhaps the model of the component needs to change. However, I'm taking a stab in the dark. If you have access to the component's author, then definately being able to add some debugging code or otherwise work out where the exception is thrown is a lot more useful than trying to deal with a black box.

Christian Graus - Microsoft MVP - C++

Reading Registry Data

TeamWild20-Oct-05 8:43

TeamWild

20-Oct-05 8:43

I am trying to find out the version numbers of the components that appear in the Add/Remove programs list.

The data I'm after is stored in the registry BUT I don't seem to be able to access it via code but I can view it usingg Regedit.

Below is the code I'm using to try and access the data. As you can see, I've tried setting the access permission but I'm not entirely sure if my implementation is correct.

If I trim the registry path down to "Software\\Microsoft\\Windows" I get a key returned.

Any ideas???

Cheers,
Ben

<code>
/// <summary>
/// Create a registry key object to allow us to retrieve installed components version numbers.
/// </summary>
/// <returns>The registry key to use if no error occured, else null</returns>
private RegistryKey GetRegistryKey()
{
RegistryKey sysSettingsKey = null;

try
{
// Setup permission to ccess the key
RegistryPermission f = new RegistryPermission(PermissionState.Unrestricted);
f.AddPathList(RegistryPermissionAccess.Read,
"HKEY_LOCAL_MACHINE\\SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Uninstall");

// Try and open the key to see if it exists
sysSettingsKey = Registry.LocalMachine.OpenSubKey("Software\\Microsoft\\Windows\\Current Version\\Uninstall");

// Was the key found?
if( sysSettingsKey == null )
{
MessageBox.Show("Unable to access the uninstalll information", "Registry Error", MessageBoxButtons.OK, MessageBoxIcon.Exclamation);
}
}
catch(Exception ex)
{
MessageBox.Show(string.Format("Exception Detected: \n{0}",ex.ToString()),"Registry Error", MessageBoxButtons.OK, MessageBoxIcon.Exclamation);
}
return sysSettingsKey;
}
</code>

-- modified at 14:43 Thursday 20th October, 2005

Christian Graus20-Oct-05 10:40