|
I hate to ask this question, but this is my first time using SQLite from the nuGet package.
System.Data.SQLite (x86/x64) and it installed EF6, SQLite Core, SQLite EF6, SQLite LINQ
I wrote a DLL for the DAL, and a DLL for the SQL Linq Call. Then I wrote a function to just get a count from the table to trigger creating the database. I tested it using SQL Server and it worked fine. Created the database and tables, then seeded the tables.
So then I switch over to SQLite. Adjusted the App.Config file - see below
This App.Config is from the main program project. I have App.Configs in the DLL's as well, not sure If I need them or not.
<configuration>
<configSections>
<section name="entityFramework" type="System.Data.Entity.Internal.ConfigFile.EntityFrameworkSection, EntityFramework, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" requirePermission="false" />
<!-- For more information on Entity Framework configuration, visit http:
</configSections>
<connectionStrings>
<add name="DefaultConnection" connectionString="Data Source=WSACORE2-VM1\SQLServer; Initial Catalog=AccountMateExtras2016; User ID=mvc.connector; Password=9Moret9z" providerName="System.Data.SqlClient" />
<add name="SQLiteConnection" connectionString="data source=d:\Amp\SIS\AMExtras.sqlite" providerName="System.Data.SqlLite.EF6"/>
</connectionStrings>
<startup>
<supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5.1" />
</startup>
<system.data>
<DbProviderFactories>
<remove invariant="System.Data.SQLite" />
<add name="SQLite Data Provider" invariant="System.Data.SQLite" description=".NET Framework Data Provider for SQLite" type="System.Data.SQLite.SQLiteFactory, System.Data.SQLite" />
<remove invariant="System.Data.SQLite.EF6" />
<add name="SQLite Data Provider (Entity Framework 6)" invariant="System.Data.SQLite.EF6" description=".NET Framework Data Provider for SQLite (Entity Framework 6)" type="System.Data.SQLite.EF6.SQLiteProviderFactory, System.Data.SQLite.EF6" />
</DbProviderFactories>
</system.data>
<entityFramework>
<defaultConnectionFactory type="System.Data.Entity.Infrastructure.SqlConnectionFactory, EntityFramework" />
<providers>
<!--<provider invariantName="System.Data.SQLite" type="System.Data.SQLite.EF6.SQLiteProviderServices, System.Data.SQLite.EF6, Version=1.0.101.0, Culture=neutral, PublicKeyToken=db937bc2d44ff139" />-->
<!--<provider invariantName="System.Data.SqlClient" type="System.Data.Entity.SqlServer.SqlProviderServices, EntityFramework.SqlServer" />-->
<provider invariantName="System.Data.SQLite.EF6" type="System.Data.SQLite.EF6.SQLiteProviderServices, System.Data.SQLite.EF6" />
</providers>
</entityFramework>
</configuration>
Then I ran this in the EF DLL and got that error in the subject line.
Imports System
Imports System.Collections.Generic
Imports System.Linq
Imports System.Data.SQLite
Imports System.Data.SQLite.EF6
Imports System.Data.SQLite.Linq
Imports AccountMate_DAL.DAL
Imports AccountMate_DAL.Models
Imports AccountMate_EF6
Public Class ef_seed
Public Shared Function Create_Database() As Integer
Dim pValue As Integer = 0
Using context As New ameContext()
pValue = context.SMTP_SEND.Count()
End Using
Return pValue
End Function
End Class
I have no clue on this, and I did search for 2 hours on the subject. Any help appreciated!
Maybe the Linq I use above is not valid with SQLite.
|
|
|
|
|
Well first off, SQLite doesn't support automatic database creation, you have to manually create the database. So I added that add-in to my Firefox and created the database.
I'm wondering if this is worth the time. This program uses old Dbase IV or Foxpro database files, and I wanted to use something faster and more modern for a new program feature that stores all the emails sent in the database, so if one fails it can be resent.
This kicked my butt, but I don't get an error anymore. I couldn't figure out the App.Config part the way the bulk of examples showed, but I did figure out how to point the database to a network drive via code in the Application, and pass it to EF6 DAL.
So in case your looking for an example of how to point SQLite to a network drive without hard coding the path, this is it, well I think it is, until something better comes along.
I apologize in advance for this being in VB. All the examples where in C#
My project has 3 modules, because it's getting too large in size.
Main Project EXE
Data Access Layer DLL
Entity DLL
I went back and stripped out the SQLite from the main project and Entity DLL and just installed it in the DAL DLL. Then just made references for SQLite back to the DAL. Then stripped out the SQLite and EF stuff in the App.Configs of DLL's
So in the DBContext class in my DAL, I added the conn string to use in New()
Public Class ameContext
Inherits DbContext
<ObsoleteAttribute("The default connection factory should be set in the config file or using the DbConfiguration class. (See <a href="http:
Public Shared Property DefaultConnectionFactory As IDbConnectionFactory
Public Sub New(ByVal conString As String) '<< I added the conn string to use
MyBase.New(conString)
'SQLite doesn't support auto database creation
Entity.Database.SetInitializer(Of ameContext)(Nothing)
End Sub
End NameSpace
Then I added a SQLiteConfiguration File to my DAL Class, which gets called from the DBContext above.
Imports System.Data.SQLite
Imports System.Data.SQLite.EF6
Imports System.Linq
Imports System.Text
Imports System.Data.Entity.Core.Common
Imports AccountMate_DAL
Namespace System.Data.SQLite.Entity
Public Class SQLiteConfiguration
Inherits DbConfiguration
Public Sub New()
SetProviderFactory("System.Data.SQLite", SQLiteFactory.Instance)
SetProviderFactory("System.Data.SQLite.EF6", SQLiteProviderFactory.Instance)
SetProviderServices("System.Data.SQLite", CType(SQLiteProviderFactory.Instance.GetService(GetType(DbProviderServices)), DbProviderServices))
End Sub
End Class
End NameSpace
And added a SQLiteConnectionFactory Class to my DAL Class, which gets called from the DBConfiguration above.
Imports System.Data.Common
Imports System.Data.Entity.Infrastructure
Imports System.Data.SQLite
Public Class SQLiteConnectionFactory
Implements IDbConnectionFactory
Public Function CreateConnection(ByVal connectionString As String) As DbConnection Implements IDbConnectionFactory.CreateConnection
Return New SQLiteConnection(connectionString)
End Function
End Class
The App.Config for the VB App, All I did was declare the DefaultConnectionfactory, which is the name of my DAL.
<entityFramework>
<defaultConnectionFactory type="AccountMate_DAL.SQLiteConnectionFactory, AccountMate_DAL" />
</entityFramework>
Finally in the program, when initializing the context, I added the connstring, connstring is that string that was in the App.Config of the vb.net app shown below, higlighted in blue. Get the string and then delete it.
<add name="SQLiteConnection" connectionString="data source=d:\Amp\SIS\AMExtras.sqlite" providerName="System.Data.SqlLite.EF6" />
Dim context = new ameContext(connstring)
Hope that helps, like I said it was pretty confusing to figure out. And the examples were so basic and assumed it works.
[edit] 05//11/2013
Had no modify the DBContext to stop Database Creation
But I wrote a record to it, and updated the record 5 times now.
Be cool if I can figure out a way to detect the whether its using SQL Server or SQLite in the DBContext.
modified 13-May-16 11:08am.
|
|
|
|
|
How we can get last 12 months using current month with we select departmentTitle and ReadingDate(Apr 2016) from table
and also add sum of meterreading using departmentTitle and ReadingDate.
My Query is:
SELECT CONVERT(varchar(3), ReadingDate, 100) AS XText,SUM(MeterReading) AS DataText,MonthlyTarget AS YText FROM tblReadings WHERE ReadingDate =apr 2016 AND DepartmentTitle=ADS GROUP BY CONVERT(varchar(3), ReadingDate, 100), MONTH(ReadingDate),MonthlyTarget ORDER BY MONTH(ReadingDate)
Required output is:
[Month from date] [Sum of meterreading]
[Apr] [12000]
[Mar] [16000]
[Feb] [20000]
[Jan] [18000]
|
|
|
|
|
Assuming you are storing your date as a date or datetime you use
where Readingdate >= dateadd(year,-1,ReadingDate)
Typed from memory so you need to check the syntax
Never underestimate the power of human stupidity
RAH
|
|
|
|
|
Thanx...
But I want selected month and previous 12 months..
and my where condition is
Where ReadingDate <= 'Nov 2016' AND DepartmentTitle='ABC'
Required OUTPUT:
Nov 2016
Oct 2016
Sept 2016
Aug 2016
Jul 2016
Jun 2016
May 2016
Apr 2016
Mar 2016
Feb 2016
Jan 2016
Dec 2015
|
|
|
|
|
Do some research into the datetime object in sql server, by using that you can get exactly what you want.
Never underestimate the power of human stupidity
RAH
|
|
|
|
|
To only include the last 12 months of data use this in your WHERE clause
WHERE ReadingDate BETWEEN DATEADD(YY, -1, GETDATE()) AND GETDATE()
As aside, instead of using
CONVERT(varchar(3), ReadingDate, 100) AS XText try (SQL 2008 and later)
DATENAME(mm, ReadingDate) AS XText
|
|
|
|
|
CHill60 wrote: SQL 2008 and later
In this case, MSDN is wrong, or at least misleading. The DATENAME function was available at least as far back as SQL 2000:
DATENAME : SQL Server 2000 Books Online[^]
"These people looked deep within my soul and assigned me a number based on the order in which I joined."
- Homer
|
|
|
|
|
 Thanks for info.
I always check when a function I'm suggesting was introduced - you're right, MSDN is misleading on this one. I'll do more robust checking next time 
|
|
|
|
|
You, like me, assumed SQL server, if the OP does not even deign to inform us as to the database he is using then I don't think the onus is on you to be specific as to the when the function became available!
Never underestimate the power of human stupidity
RAH
|
|
|
|
|
Thanks ....
We want only required Month and Year from date instead of date in where condition.
This is my Query:
SELECT TOP 12 CONVERT(varchar(3), ReadingDate, 100)+CONVERT(varchar(4), ReadingDate,102) AS XText,SUM(MeterReading) AS DataText,(MonthlyTarget) AS YText
FROM tblReadings
WHERE DepartmentTitle='abc'AND ReadingDate BETWEEN DATEADD(mm,-12,ReadingDate) AND '31 may 2016'
GROUP BY CONVERT(varchar(3), ReadingDate, 100)+CONVERT(varchar(4), ReadingDate,102), MONTH(ReadingDate),YEAR(ReadingDate),MonthlyTarget
ORDER BY YEAR(ReadingDate) DESC, MONTH(ReadingDate) DESC
I Want this output -
'may 2016'
|
|
|
|
|
Is there any way to find out the status of opening a recordset in VBA? I thought it was the SQL that was slow but it is instantaneous. It is the open of the recordset that is slow and I would like to be able to monitor its progress.
ORIGINAL QUESTION SHOWN BELOW FOR HISTORY
I'm running an SQL query in VBA in Excel to return a record set. Is there any way to know the progress of the query so I can show it on the status bar (or elsewhere)? If there are millions of records that would be better than "please be patient" or "processing 15,000,000 records".
modified 5-May-16 19:04pm.
|
|
|
|
|
There is no status information returned by the SQL Server while the query is running, so no.
|
|
|
|
|
If you are processing 15,000,000 records in VBA you have shot yourself already, feedback is not going to save you.
That is an outrageous number to be dealing with, especially with VBA.
Never underestimate the power of human stupidity
RAH
|
|
|
|
|
That would be an extreme example. Generally the max might be in the range 300,000 records and it whips through that in about 10 seconds. It might go to a million or more. Probably it would never hit 15 million, and I agree that number would be outside reasonableness for the tools used.
|
|
|
|
|
You're still screwed.
No database engine reports back any notification of it's progress. If the query is taking 10 seconds, you won't know anything about it for 10 seconds.
The problem with doing this in VBA is that it doesn't support threading and the UI will be blocked (not able to update) while the query is running.
The only time you're going to be able to get any progress information is if you're returning records from a Reader, you read one record at a time from the result set. The problem with this is the Reader has no idea how many records it's returning. It just keeps reading records until the end of the dataset shows up from the database.
|
|
|
|
|
I have practically no experience with VBA so I can't guarantee that the following would work but I assume it will (maybe someone else can comment on this). Also I'm deliberately not addressing the issue with the high amount of records you want to deal with and the issue with the unresponsive UI. This is a kind-of workaround to be able to display progress-information:
1) Issue your query but modified to return only the count of found records.
2) Divide the record-count into batches of some size that suits you (e.g. 5000) and issue your query but with added paging-instructions: sql server paging - Google Search[^]
After one batch/page has been retrieved, you can update the progress information, then retrieve the next batch/page, until finished.
The price you're paying for the benefit of progress information is more overall time required because the database server has to execute the query multiple times. That's why the batch/page size shouldn't be too small (while a smaller size would improve the resolution of the progress update).
If the brain were so simple we could understand it, we would be so simple we couldn't. — Lyall Watson
modified 9-May-16 11:38am.
|
|
|
|
|
The answer is the same for 300.000 or 15.000.000 records.
DON'T DO IT.
Think about it, do you need to handle 300.000 records for the dialog with user ?
If you need to update your 300.000 records, see if it is possible to device a sophisticated SQL commend that will do it on server and avoid transfering the records.
You should tell the reason of the need of 300.000 records, there is certainly a solution that don't need such transferts.
Patrice
“Everything should be made as simple as possible, but no simpler.” Albert Einstein
|
|
|
|
|
OK - well - it works. The user doesn't see the records, all I do is ask the user between what dates then I use that as input to the SQL. So they get a slice of the file. I write that out then I do a join to a third file and write out that file. That end file is used as input for a pivot table. Therefore except for initially asking the date range, there is no user presentation except via the pivot table which is extremely fast because by then I have trimmed all the fields I don't want and all the records I don't want.
Originally I was trying to do this as SQL in Excel itself through the SQL Editor but note well that if you try to have more than two tables in joins in a query it will not work. (I posted this in SQL questions already). That is why I went to VBA in the first place. So I open a box, get the date from the user, process the records, then refresh the pivot. It seems to work well. Technically I -might- be able to combine the two queries but really for clarity sake it's almost better as 2 queries because I do field reformatting and so on. Yes doing two writes instead of one is slower so I may check this out later however I have something that works. I am of course open to suggestions how could it be better - but I don't see a horrible performance issue (yet). I think what happens is when you set a recordset equal to an SQL statement, it doesn't really run it until you try to open the recordset. Which would kind of make sense, I guess.
Thanks for all help, many thanks.
|
|
|
|
|
Also.... if I could perform SQL statements on a recordset that would be ideal. That would then allow me only 1 write. I don't think you can do this but if anyone does do it I'd like to know. It would also reduce a bit of code where I am manipulating things also in VBA before I do the write, which could be done in the SQL statements more simply.
|
|
|
|
|
You should show you SQL code.
SQL is veru powerful and can do many things without intermediate database writing.
Patrice
“Everything should be made as simple as possible, but no simpler.” Albert Einstein
|
|
|
|
|
myQuery = "SELECT *, Format([End Time],'YYYY-MM-DD') AS EndTimeDate from CDRecords.csv " & " WHERE Format([End Time],'YYYY-MM-DD') BETWEEN '" & searchStartDate & "' AND '" & searchEndDate & "'" & " ORDER BY [End Time] ASC"
myQuery = "SELECT *, R.[AreaName] from DateSlicedCDR.csv AS Q LEFT OUTER JOIN AreaCodes.csv AS R ON Q.[AreaPrefix] = R.[AreaPrefix] ORDER BY [EndTimeDate] ASC"
Those are the two queries. They work fine. NB: Formats have to match on the dates.
searchStartDate and searchEndDate variables are properly formatted dates obtained from the user form.
|
|
|
|
|
Not serious !
csv files gave never been databases.
You need to use real databases, bt setting indexes, you will get dramatic improvement.
Patrice
“Everything should be made as simple as possible, but no simpler.” Albert Einstein
|
|
|
|
|
Well.... it comes in text file format, so, it would be yet another step to convert it to something else, but I could do that. ?What would you suggest, if I did do that?
Performance isn't horrible right now but definitely yes I'd like to speed it up plus as numbers of records get higher it will get worse.
Use currently appends the text file with new data, so chances are the first file I get will be in text format - however - looking to the future, yes, it could be way better. I'm listening...
|
|
|
|
|
Situation is complicated, would need to know full details on data flow to have global view and give correct advices.
All I can say is that actual situation resort to brute force and need heavy changes to optimize.
This is getting by far out of scope of this forum.
You need to hire a professional.
Patrice
“Everything should be made as simple as possible, but no simpler.” Albert Einstein
|
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
