Click here to Skip to main content
15,397,981 members
Home / Discussions / C#
   

C#

 
GeneralRe: Linq to SQL With Enum On Null Field Pin
OriginalGriff2-Jul-22 9:52
mveOriginalGriff2-Jul-22 9:52 
GeneralRe: Linq to SQL With Enum On Null Field Pin
Kevin Marois2-Jul-22 9:59
professionalKevin Marois2-Jul-22 9:59 
QuestionRe: Linq to SQL With Enum On Null Field Pin
Eddy Vluggen2-Jul-22 10:37
professionalEddy Vluggen2-Jul-22 10:37 
AnswerRe: Linq to SQL With Enum On Null Field Pin
Kevin Marois2-Jul-22 14:58
professionalKevin Marois2-Jul-22 14:58 
AnswerRe: Linq to SQL With Enum On Null Field Pin
Dave Kreskowiak2-Jul-22 4:59
mveDave Kreskowiak2-Jul-22 4:59 
GeneralRe: Linq to SQL With Enum On Null Field Pin
Kevin Marois2-Jul-22 9:43
professionalKevin Marois2-Jul-22 9:43 
AnswerRe: Linq to SQL With Enum On Null Field [MODIFIED] Pin
Richard Deeming4-Jul-22 21:40
mveRichard Deeming4-Jul-22 21:40 
QuestionStreaming data from one datalayer to another Pin
Evilfish200029-Jun-22 4:14
MemberEvilfish200029-Jun-22 4:14 
I have a problem where I am creating a data layer for a project, where I have a scenario I don't really know which solution is better. The main function of the project/service is to regularly check for changes in several databases and copy the information to a data warehouse(one big database really).

One scenario is when you add an new table to be tracked in the system. It has a specific task to copy the entire table from the site database to the warehouse. It does this by opening a data reader and then generate a batch INSERT script until it reaches a limit and then it execute the script towards the data warehouse. In pseudo code it looks like this:
C#
using (IDbCommand clientCommand = Connection.CreateCommand())
{
    clientCommand.CommandText = clientSql;
    using (IDataReader clientReader = clientCommand.ExecuteReader()
    {
        while(clientReader.Read())
        {
            AddDataToList(clientReader);
        }

        if(List.Count >= 100)
        {
            using (var com = warehouseConnection.CreateCommand())
            {
                com.CommandText = GetInsertScript(List);
                AddedCount += com.ExecuteNonQuery();
            }
        }
    }
}

Its more complicated than that, with more checks and sql code. I see two main issues with this:

1: Database logic in business logic that I want to move over to my data layer. This will make the code testable and easier to understand.

2: I do not like that we wait for data from the client, and then wait for data to be written to the other database, when this could be done in 2 tasks that transfer and receiving information asynchronously with each other via ConcurrentQueue for instance. This speeds up the process, which is also an issue in some cases.

The problem I am having is that I that I cant get all the data in one go and then add it to the warehouse. Sometimes the tables are huge, which can lead to memory issues on the service. So I want to process the data as they come in through the client reader and continuously add them to the warehouse. But, I also want the database logic in the data layer. Adding the data is simple, but retrieving it is a bit more complicated. Researching this issue have given me 2 choices:

Option 1 (Pull from data layer):

I could use the yield keyword to return data as it comes through the reader in my data layer:
C#
public IEnumerator<object[]> GetTableContent()
{
    SqlCommand command = new SqlCommand("command", SqlConnection);
    using (var reader = command.ExecuteReader())
    {
        var items = new object[reader.FieldCount];
        reader.GetValues(items);
        yield return items;
    }
}

With this I can use it in a foreach loop, and each time I get a return I add it to the queue. Another task takes from the queue and adds it to the data warehouse. With this I can control the when I want a new result from the site database and not add to many items to the queue if the data warehouse is busy.

Options 2 (Push from data layer):

I can use Actions to push data to the caller:
C#
public void GetTableContent(Action<object[]> dataAction)
{
    SqlCommand command = new SqlCommand("command", SqlConnection);
    using (var reader = command.ExecuteReader())
    {
        var items = new object[reader.FieldCount];
        reader.GetValues(items);
        dataAction.Invoke(items);
    }
}

Here I call the method and it will start to give me information through an action instead.

From the 2 options I find the pull method is a little more graceful and it is more inline with how the reader works. But I have also read that people are having issues with this where having yield returns in using have been an issue. That's where option 2 has been suggested. It should be be able to do the exact same thing. So... which option is better. Option 1 or Option 2 or maybe there is another way I haven't even considered?
AnswerRe: Streaming data from one datalayer to another Pin
Pete O'Hanlon29-Jun-22 20:46
mvaPete O'Hanlon29-Jun-22 20:46 
PraiseRe: Streaming data from one datalayer to another Pin
Evilfish200029-Jun-22 21:40
MemberEvilfish200029-Jun-22 21:40 
QuestionC# voice communication between 2 local pcs Pin
Getyourwings UK28-Jun-22 10:07
MemberGetyourwings UK28-Jun-22 10:07 
QuestionIs there anything that Python or Java can do for windows GUI applications while C# can't ? Pin
Code4Ever24-Jun-22 2:33
MemberCode4Ever24-Jun-22 2:33 
AnswerRe: Is there anything that Python or Java can do for windows GUI applications while C# can't ? Pin
Craig Robbins24-Jun-22 6:33
MemberCraig Robbins24-Jun-22 6:33 
AnswerRe: Is there anything that Python or Java can do for windows GUI applications while C# can't ? Pin
Richard MacCutchan24-Jun-22 22:38
mveRichard MacCutchan24-Jun-22 22:38 
QuestionLinq TO SQL DistinctBy Question Pin
Kevin Marois23-Jun-22 18:16
professionalKevin Marois23-Jun-22 18:16 
AnswerRe: Linq TO SQL DistinctBy Question Pin
OriginalGriff23-Jun-22 19:49
mveOriginalGriff23-Jun-22 19:49 
GeneralRe: Linq TO SQL DistinctBy Question Pin
Richard Deeming23-Jun-22 22:36
mveRichard Deeming23-Jun-22 22:36 
AnswerRe: Linq TO SQL DistinctBy Question Pin
Richard Deeming23-Jun-22 22:43
mveRichard Deeming23-Jun-22 22:43 
QuestionDrawing with c# Pin
Member 1557095222-Jun-22 23:08
MemberMember 1557095222-Jun-22 23:08 
AnswerRe: Drawing with c# Pin
Richard Deeming22-Jun-22 23:55
mveRichard Deeming22-Jun-22 23:55 
GeneralRe: Drawing with c# Pin
Member 1557095223-Jun-22 0:08
MemberMember 1557095223-Jun-22 0:08 
GeneralRe: Drawing with c# Pin
Richard Deeming23-Jun-22 0:42
mveRichard Deeming23-Jun-22 0:42 
AnswerRe: Drawing with c# Pin
Gerry Schmitz23-Jun-22 3:09
mveGerry Schmitz23-Jun-22 3:09 
AnswerRe: Drawing with c# Pin
OriginalGriff23-Jun-22 3:59
mveOriginalGriff23-Jun-22 3:59 
AnswerRe: Drawing with c# Pin
Member 1557095223-Jun-22 21:16
MemberMember 1557095223-Jun-22 21:16 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.