Re: general advice on starting to work with DB programming ? - Design and Architecture Discussion Boards

You don't need an intermediary app, besides all that would do would eat memory on the server, which I'm sure SQL Server would prefer to eat Smile | :)

I think in general the issue is that because you'll have a POCO with a collection of another POCO, people will typically just use that collection to get the related details, without realising that in the background these POCO's are likely mapped to tables, and you've just asked EF to do SELECT * from two tables Smile | :)

But you can use linq queries to narrow down what you want and EF will then just bring back just what you want.
Its just 'harder', or at least less obvious that you need to do that. And its pretty easy to see the SQL EF generates, and have it logged so that you can check it.

I find that getting the state of EF object graphs the harder thing to grasp and visualise.

I appreciate your thoughts, and the take-away I have from your comments is to familiarize myself with EF.

thanks, Bill

«I'm asked why doesn't C# implement feature X all the time. The answer's always the same: because no one ever designed, specified, implemented, tested, documented, shipped that feature. All six of those things are necessary to make a feature happen. They all cost huge amounts of time, effort and money.» Eric Lippert, Microsoft, 2009

BillWoodruff wrote:
if there isn't a way to have an "intermediary" app running on the server that takes Linq queries from clients as input, and returns highly-filtered data.

Yes but then all you are doing then is introducing yet another server that might have to deal with too much data.

Instead one should start with a design that applies the filter in the database and does it in an effective manner. Doing that reduces the load on the server and the data that needs to be returned.

Thanks for your reply, and I think I grok the very common-sense gist of your comment, which I interpret as "memory ain't cheap on the server, either." Hope that's not too far off-the-mark. Bill

«I'm asked why doesn't C# implement feature X all the time. The answer's always the same: because no one ever designed, specified, implemented, tested, documented, shipped that feature. All six of those things are necessary to make a feature happen. They all cost huge amounts of time, effort and money.» Eric Lippert, Microsoft, 2009

It isn't just memory - moving the data to another server requires work both for each server and the OS as well. And that is true even on the same box.

One of the worst solutions I have seen for applications is with designs that decided they were going to make the app database agnostic and the way they did that was to move all (all) of the business logic off of the database. That works for small volumes but is absolutely useless for large volumes when large volumes must be processed (in one case I saw they moved the entire database to a client box, processed it, then moved it back.) It couldn't scale at all. Probably could have scaled but they didn't design it that way. And processing on the database even if didn't scale for massive volumes at least would have work for the real volumes that their actual solution couldn't handle.

The ORM I have worked with yet (DataObjects.Net, code-first) allows you to specify which referenced entities / entity-collections you want to have instantiated, e.g.:

C#

var query = (from customer in Query.All<Customer>()
             where customer.Something == true
             select customer)
            .Prefetch(customer => customer.Orders);

I assume there's something similar in Entity Framework or NHibernate. This should cover the main point of your question regarding the "glut"..?

- Sebastian

Thanks, I will check out DataObjects.NET !

«I'm asked why doesn't C# implement feature X all the time. The answer's always the same: because no one ever designed, specified, implemented, tested, documented, shipped that feature. All six of those things are necessary to make a feature happen. They all cost huge amounts of time, effort and money.» Eric Lippert, Microsoft, 2009

My first reaction when you say that you primarily want to work with C# and Linq, is that you should take a look at RavenDB[^]. I wouldn't say it's a mature product as it has a fair share of gotchas, but it's made for C# and linq.

But then I read that you want to work with multiple-references/linkages across categories/objects, and then there's no shortcuts to take anymore. You'll need to learn a proper RDBMS (or maybe a graph database as you already investigated). And the vast majority of those use SQL.

As a first read I'd recommend Sanders article[^], it's covering the query language pretty well in a structured manner while still being easy to read. DML, DDL and indexing can wait for later.
As you're coming from the third generation programming world you'll now you probably notice that SQL is a fourth generation language. You need to change your approach to programming and stop telling the program what to do, but rather telling it what you want. You also need to start thinking in sets rather than rows or objects. So you need to brush off your set theory[^] knowledge

As a second step I'd recommend to learn normalization[^]. Since the whole concept of normalization reeks of buzzwords, that makes the whole concept rather hard to grasp for a layman, I'd actually recommend an article that shows how to do it instead[^].

If I should address the discussion you refer to, I should say that one needs to use the right tool for the right job. And when people fail it's not on choosing the tools but rather on defining what the job is.
What a database is good at is to Store, Retrieve, Filter and Aggregate data, and it's normally much better than anything you can think up in the business layer, it's optimized for exactly that. But that's about it. (If I may oversimplify it)
Math's, presentation, formatting and more or less everything else should be done somewhere else.

A common problem with ORM layers is that people tend to do the filtering and aggregation in the business layer, sending huge amounts of data forth and back between the layers. This is not a problem with ORM, it's an architectural problem and people not knowing how to use the tools. Putting the business layer in the database is just as wrong.

The most common example is the N+1 problem. Assume you have a Master Detail grid where the master has 1000 rows. Most ORMs works well when you use lazy loading but if you've done it wrong and load all the Details at once you end up with 1001 queries to the database. If you use lazy loading it fetches the data for the detail when the user clicks on it and the chance that the user will click on all detail views are usually pretty small, but in this case you should use eager loading and create the SQL yourself.

Wrong is evil and must be defeated. - Jeff Ello

I really appreciate this very thoughtful answer, and will study it carefully. As I have studied the architecture and "gestalt" of Neo4j, I find I am very attracted to the model and its functionality, but do not want to get involved with all the graph visualization facilities. I discovered Neo4j while reading one of Marc Clifton's visionary articles here: [^].

I am a big fan of Mehdi Gholam's work, and have played with his RavenDB a bit. Long ago (1980's I took lecture-notes for a lecture-note-service for a young U.C. Berkeley Computer Scientist and lecturer, Mark Tuttle, who worked with Stonebraker (who later created Ingres and PostGres) on relational DB's, so I have had some exposure to "normalization," but not enough !

Since I am not constrained by financial considerations, I do have the time to spend on pursuing "out there" DB's, where relationships go beyond is-a and has-a.

Update: I have installed, and am studying, and using, the triple-store .NET graph-db BrightStarDB. Their excellent documentation and how-to's made it very easy to get a basic DB working (very refreshing, that experience).

My take-away from your very educational response is that I should do some basic study of SQL, first.

thanks, Bill

«I'm asked why doesn't C# implement feature X all the time. The answer's always the same: because no one ever designed, specified, implemented, tested, documented, shipped that feature. All six of those things are necessary to make a feature happen. They all cost huge amounts of time, effort and money.» Eric Lippert, Microsoft, 2009

modified 15-Feb-15 4:17am.

I believe you're thinking of RaptorDB, I meant Ayendes ravendb.net[^] which is a different beast.

BillWoodruff wrote:
My take-away from your very educational response is that I should do some basic study of SQL, first.

I find it more important to learn normalization. The why's and how's.
Before one works with denormalized databases, one needs to know when and how to cut the corners without messing everything up.

But if you're into Graph databases this is not a problem. Normalization is quite built into them.

What in an RDBMS would be the relation between two otherwise (usually) pointless (as in not carrying any information) IDColumns in the related tables, is in a GraphDB an edge connecting two vertices, an entity of its own.
But still just two ways of doing the same thing. Normalization rules still applies.

BTW, Thanks for mentioning BrightStarDB, I didn't know about it.

Wrong is evil and must be defeated. - Jeff Ello

update: Scott Hanselman has a PodCast about BrightStarDB with founder Kal Ahmed (August, 2014) here: [^]

Hi Jorgen, thanks for your additional suggestions; I will check out RavenDB, and take your advice on focusing on studying RDBMS normalization, first ... what memories I have of Tuttle's brilliant lectures are rather fogged-in by time after nearly thirty-years Smile | :)

I am impressed by BrightStar (written by C# programmers for C# programmers), and wish I had the technical depth/background to compare it to Neo4j meaningfully, but the company has done a great job of providing detailed C# sample programs.

Neo sponsors a free download of the recent book, "Graph DataBases" from O'Reilly, which is an excellent resource that does cover fundamentals of Graph Databases: [^]. As someone with over forty years of experience in technical editing, and writing, I can say that this book is a model of clear, well-organized, technical prose. Unsurprisingly, principals of Neo4j are among the authors.

cheers, Bill

«I'm asked why doesn't C# implement feature X all the time. The answer's always the same: because no one ever designed, specified, implemented, tested, documented, shipped that feature. All six of those things are necessary to make a feature happen. They all cost huge amounts of time, effort and money.» Eric Lippert, Microsoft, 2009

modified 16-Feb-15 4:48am.

Hello.
First, let me introduce myself:
I am a Java (6 years exp), Python(2 years exp), and C (2 years exp) developer.

This thread is both a question for all programmers (feel free to reply!) and a little bit informative for some Smile | :)

So, I have been working on a relatively complex system (link[^]) that does the following:

a- (OpenCV) reads frame by frame and detect motion, from a webcam
b- reads from a second webcam without processing, just to watch
c- controls a parallel port or usb device
d- generates custom simple sounds
e- logs everything described here.
f- has a Graphic User Interface to control all of the above parameters.

So 2 webcams, audio out, usb device, and graphic user interface, is in short what this system has.

I've done this in python and it took me some 10k lines of code and weeks/months of time. So far works with some random errors, and a below-decent (in my opinion) interface.

I wanted to ask this community if Java might be a better language at accomplishing this task, which is my hypothesis, but would like a second opinion or your general opinion about this:

1- OpenCV
2- Generating audio
3- Controlling a USB device
4- multithreading
5- GUI

-------

I don't want to bias your opinion, but so far here is what i currently believe based on my work and exp programming on both languages:

1- OpenCV: Probably the same in both languages. Maybe Java is easier/more mature but IDK
2- Audio: Probably the same (someone might have done an audio class before)
3- Low level peripherals: I don't know. Probably Java has a disadvantage here?
4- Java's multithreading and multiprocessing seems to be far superior. Here's why:
. When you launch threads in Java, the JVM will allocate it on a free processor if available. If you have 8 processors and you launch 8 threads, ideally it will allocate one in each.
. While in Python , multithreading will use the same process for all threads. I managed to work around this by using multiprocessing [^] and what i gained was to make use of quad core and some better stability (OpenCV would complain with TKinter running at the same time, stability issues if using just one process, believe me).
. In Java, launching many threads will continue letting you use all variables in the "main program" ; it might be bad practice but it will let you anyway
. In Python, the main program must open joblists and pipes to send / receive information from the (many) modules and processes running simultaneously. This bloated my original code a lot (classes that had 200 lines before, now have 300 or more to handle jobs and queues, etc) , but was absolutely necessary because using just one process wouldn't have worked (i tried).

5- GUI:
. Java: i would have used JFrame, JPanel, and all that Swing package, which would look nice and behave properly
. Python: I used Tkinter AND also Glade GTK
I used both because Tkinter comes by default with Python but looks horribly (and behaves somewhat bad when interacting with so many processes . "Segmentation fault" Mad | :mad:

).
So i introduced a second GUI: Glade GTK. Which looks good and behaves better than TK but it needs additional packages installed.

I'm in no way ranting; i even almost finished my Python code, but i am just wondering if i have done it in Java would have been better ?

So what would you use? Python ? Java? Any other language?
Sorry for the long post! cheers !

Just a programmer.

For a desktop based UI, I would use C# and WPF. You can drop to low level code with it if you need to. It plays well with things like OpenCV and it is a powerful way to build custom UIs. The downside is that it's complex to learn.

I have a table called Lookups. It's used mainly to populate ComboBoxes. There's a button next to it that opens an editor where the user can add, edit, and remove lookups. The table has a column called "LokupType".

Say for example the user enters 3 lookups called "Employee Type" and they are "Manager", "Driver", and "Dispatcher". The Employee screen has an Employee Type combo and editor button. and for employees the LookupType is "empytype", and that is hardcoded in the editor button's click routine. I can hard code it here because the code is in the Employee Screen.

In the Employee screen the employee is assigned a type from a combo loaded by a query that uses the LookupCode of "emptype". In other words, there's a query that returns only employee type lookup items.

So far so good.

Now here's the problem... In another screen I need to populate a list of only those employees who are Dispatchers.

Because the lookups can be removed, how can I know for sure which employees are Dispatchers?

In the past I used another column called SystemCode, and I would hard code in values such as:

LOOKUPTYPE   CAPTION       SYSTEMCODE
=========================================
emptype      Manager       empmanager
emptype      Driver        empdriver
emptype      Dispatcher    empdispatch

Because I had the system code, I could pull only those employees who are dispatchers. Now, the requirements call for ALL lookups to be editable and removable.... so there's no SystemCode.

I'm open to suggestion.

If it's not broken, fix it until it is

There's currently a small internal debate over what language we should be writing an email polling service.

It is a part of a much broader system. Various accounts will be set up, and emails will be pulled from them and then encrypted and stored in a database.

As it is part of a larger system we are planning on using an ORM such as Entity Framework and have a DAL or C# API to handle all things database. C# .NET has a solid networking API and some mail related stuff already, as well as solid encryption.

Concerns surrounding using C++ include;

- Having to write network code from scratch (limited on external library use)
- Can not use Entity Framework or the C# database API, would require database code to be repeated in 2 languages allowing for transfer of bugs, more time taken etc..
- Encryption.. The clients are going to be .NET and so we want to use .NET encryption from start to finish so that encryption is standardized throughout
- Later in the projects life we want to support more than just Pop3 and IMAP, Microsoft provide a managed API for it's exchange web services. To achieve this in C++ is not elegant and not recommended by Microsoft

What are the advantages of using C++ in this scenario if any?

Given all the reasons you have listed for not using C++, why are you even considering it?

The reason it's being considered is because one senior developer is adamant it needs to be in C++. the rest of the team is set on C# however, he can't put together a coherent argument for C++ other than memory management and I'm trying to assess the situation.

Given that he's the one who's adamant that it should be C++, why are you looking for reasons? If he cannot articulate a coherent argument in favour, he deserves to lose here.

Member 8456971 wrote:
he can't put together a coherent argument for C++ other than memory management

But C++ does not provide memory management, other than allowing you to consume as much as you want. If you do not win this argument then there is something seriously wrong with the management of your company.

I would like to a continuous discussion about my question posted here:

How do I load a UI based on a user specific role[^]

and as was suggested I post a question here.

I am now currently reading suggestions on the answer posted on that question. If in any case you also have other sources please feel free to tell me

Thanks in advance Big Grin | :-D

-----

Update:

I would prefer using winforms on this project for im more comfortable with it. I usally develop using VS2012 with MSSQL2008r2 for my database. (I havent had the chance to upgrade yet).

And here's the complete content of the link (as suggested):

"This is just an educational query wherein I am trying to accomplish an application that loads a specific UI (after login screen) depending on a role assigned to a user.
I just need materials to study and read on, I have been searching google and I found this article so far

PluginManager - A C# utility to load plug-in based components[^]

Its almost close to what I am trying to study and accomplish, only that it does not full describe how the plugin is created and how to create the plugin manager.

As I said I am trying to accomplish an application that loads specific user interface depending on a role assigned to a user. The reason behind is that each user of the app has a totally different job than the other, its more like a permission based app not just for security but also for a good UI design specific to a role.

So I am only asking for references regarding such matter, any article or code samples will be very useful. I am still looking for sources and other samples, but I think getting a more personal suggestions would be best. "

modified 29-Jan-15 20:36pm.

PhyxeTernida wrote:
I would like to a continuous discussion about my question posted here:

Why?

If you want to load something at runtime, that would be a question. Why does it have to be a discussion? And what have you tried?

Bastard Programmer from Hell Suspicious | :suss:

If you can't read my code, try converting it here[^]

it would be a discussion for as i read your comments i might have additional questions about this topic. also it does not have to be on run time perse at the moment i have multiple projects (and they are ready for deployment) having the UI that i need to load depending on a user role. my ultimate goal would be to place all of those little projects i made to a single app and load the needed UI after the user with a specified role log in.

PhyxeTernida wrote:
my ultimate goal would be to place all of those little projects i made to a single app and load the needed UI after the user with a specified role log in.

So what is your actual question?

im very sorry if om not that clear, my question would be: How am I going to accomplish that? Also I wanna know how the process work and if its possible with C# VS2012, so I am also asking for sample projects and references where I can learn as much as I can on doing such thing.

PhyxeTernida wrote:
How am I going to accomplish that?

By designing and writing some code.

PhyxeTernida wrote:
I wanna know how the process work

Not sure what process you are referring to.

PhyxeTernida wrote:
if its possible with C#

Most likely yes.

If you are looking for sample projects then you should go to the Articles section[^] and search there. When you have a specific question then come back here and people will try to help you.