|
I would like to know which of the following options will produce the fastest transactions (and why). Unfortunately, I don't know enough about the low-level "nuts and bolts" of database operation to really have a good intuition on this.
My PHP script resides on one server, my MySQL database resides on another. I believe that by performing operations on a regular MyISAM table, I am incurring time costs related to both 1) communicating between the web server and the database server, and 2) performing disk operations because of MyISAM is disk-based storage engine. I do not, however, know how much of the time cost is associated with each of these factors or if both are really significant (in the sense of important).
I have some temporary data that I want to manipulate on a per-session basis, and the way I see it I have two options:
I can create a local memory database and create a table there.
$wm = sqlite_open(':memory:', 0666);
sqlite_query($wm,"CREATE TABLE patterns (pattern varchar(500))");
Or, I can create a table in memory on the existing MySQL database.
mysql_connect("not.localhost.com", "admin", "example");
mysql_select_db("example");
mysql_query("CREATE TEMPORARY TABLE patterns (pattern VARCHAR(500)) ENGINE=MEMORY");
Which one of these will be faster to perform queries on? Will there be a noticable difference? My intuition is that the second will be slower because it is a transaction on a non-local database, but I don't know this for sure. When the table is in-memory, which computer's memory is it actually in? How much volume would have to be going on for a difference to actually be noticable?
Any thoughts would be very much appreciated.
--Greg
|
|
|
|
|
GregStevens wrote: Any thoughts would be very much appreciated
I think you should take measurements, and note down the version of the database that you tested.
GregStevens wrote: Which one of these will be faster to perform queries on?
I'm hoping Sqlite, since it could be a fast implementation in native code without much overhead.
GregStevens wrote: My intuition is that the second will be slower because it is a transaction on a non-local database, but I don't know this for sure.
How about putting it one the same machine?
You also could consider Times Ten[^] from Oracle.
GregStevens wrote: When the table is in-memory, which computer's memory is it actually in?
That depends on the specific implementation; most of the time it will be in memory that's managed by Windows, putting it into the virtual memory area. Others might have optimizations.
GregStevens wrote: How much volume would have to be going on for a difference to actually be noticable?
You can review the amount of free memory using the Task Manager;
Your computers fysical memory - used memory = free memory
Once the system starts to page out memory, you'll notice delays. That can range from minimal delays (say, served from the 64Mb buffer of memory in your harddisk) to large delays (Windows reshuffling a lot on disk, paging in and out other applications that are also running, your computer nearly grinding to a halt)
Bastard Programmer from Hell 
|
|
|
|
|
Hi
I understand what index seek means - but not sure about distinction between Index Seek, RID Lookup (Heap based what the hell...) and Key Lookup (clustered index lookup)
From this reference: [^]
RID Lookup is a form of a bookmark index lookup on a table without a clustered index (a heap).
> So both clustered and non-clustered is on the disk and SQL server actually build another index in memory/heap?! And all RID's (Row ID) are stored in volatile memory!?
key lookup is a bookmark lookup on a table with a clustered index.
> So how is it different from Clustered Index Seek?
I'm a bit unclear on seek vs lookup, but seems like "non-clustered seek" (non-clustered ONLY)[^] fetches only the RID, database engine still need to perform a "lookup" by RID to fetch the actual values of selected columns.
For "clustered index seek", leaf is actual data so there should never be lookup for rows fetched via "clustered index seek" (thus no idea what a "Key Lookup" is about)
Similar question to http://www.web-shelf.com/microsoft.public.sqlserver.relationalserver.performance/index-seek-with-lookup-vs-index-seek-with-include-thread-ID1202.aspx[^]
Adding to confusion, from MSDN[^]- so RID is in heap (where? All rows RID in heap/memory!?) and Key lookup is done via clustered index on the disk? HEAP (aka "Base Table) is not Clustered Index which is stored on disk.
RID Lookup is a bookmark lookup on a heap using a supplied row identifier (RID). The Argument column contains the bookmark label used to look up the row in the table and the name of the table in which the row is looked up. RID Lookup is always accompanied by a NESTED LOOP JOIN.
Key Lookup operator is a bookmark lookup on a table with a clustered index. The Argument column contains the name of the clustered index and the clustering key used to look up the row in the clustered index. Key Lookup is always accompanied by a Nested Loops operator. If the WITH PREFETCH clause appears in the Argument column, the query processor has determined that it is optimal to use asynchronous prefetching (read-ahead) when looking up bookmarks in the clustered index. For more information about read-ahead, see Reading Pages.
Thanks
dev
modified on Tuesday, July 5, 2011 11:46 PM
|
|
|
|
|
I think that you've been thrown off by the meaning of the word "heap". Although it usually means what you think (i.e. an in-memory data structure[^]), in the context of the discussions that you quoted it means a heap-based table[^].
Anyway, Key Lookup is logically identical to RID Lookup. Physically, there are significant differences, but nine times out of ten you can count them with the rest of unimportant implementation details.
|
|
|
|
|
Many thanks
I understand the word "Heap" as I am a programmer - but how is it possible for SQL server to put in heap/memory RID's of all these million rows tables!
dev
|
|
|
|
|
Hi guys. This post is building upon the solution from my previous post in here on Jun 29th.
Synopsis: Building a scheduling/routing application on top of a data-driven app that lives in SQLServer that identifies customers that have dropped our service for "Winback" call center campaigns (and the like)
The logic for each of the campaigns is encapsulated within a query run within a stored procedure that is located on our Reporting database (it runs there because Reporting has the $**T indexed out of it) The stored procedure is executed from our local production database across a server link. My application knows absolutely nothing about the underlying stored proc. It knows of a command that it is wishing to execute for a particular campaign, and I have a mapping table that stores a pointer to the remote stored proc. I have a brokerage entry point exposed as a stored proc to my application that accepts parameters, dips into the local Mapping table, pulls the name of the stored proc, executes it across the database link, receives the result set, generates a GUID for the batch, enters the results into a local table called "Solicitations" with the GUID as a correlator and returns the GUID to the calling client.
...I was proud that I was able to accomplish all of that. I am nowhere *NEAR* what one would call a "DBA" and all of my "education" is in Oracle9i. I'm currently "working" in SQL Server 2008R2.
But now: All of the previous work has been the "sourcing" side of our process. Being able to get the data out of Reporting and into our local table is just the data pump that allows our campaigns to run. After the calling client receives the GUID as its return, I need to pass the GUID into an SSIS package that can receive that GUID, identify the campaign that is being run, dip into the mapping table to pick up the filename associated with that campaign, pull the relevant rows out of the Solicitations table, map them to a canonical format, and FTP them to an FTP server that acts as an integration point between our Customer/Account system and the system that manages call routing for our call center.
I have created an SSIS package that has a Data Flow task that Flat File Exports the Solicitations table to csv as "testy.csv" on my local file system and then an FTP task that picks that resulting file up and :21's it to the integration point. I went to the server and picked the file up and it was the file I was expecting. All of the guts of the "happy path" flow are there.
My actual question:
What I am needing to do is:
1) configure the working Data Flow Task such that it can receive a GUID and select items from a view where the correlator in the table matches the GUID for the batch.
2) I'm also needing to create a task that receives that GUID and retrieves the OutputFilename from the Campaign Mapping table so that it knows which file to pick up and transport.
3) I'm needing to modify the FTP Task such that the filename is the variable set in (2) <-I'm thinking #3 is going to be the easy part.
Has anyone ever done this before? Can anyone provide an assist on how to accomplish 1-3 above?
Thank you very much!
"I need build Skynet. Plz send code"
modified message subject on Wednesday, July 6, 2011 1:24 AM
|
|
|
|
|
I found an answer to my question. I created a stored procedure that accepts the GUID as a parameter and within the proc, I (create, add, and kick off) a job that executes the SSIS package as its only step, passing the GUID into the command line args of the command property of sp_add_jobstep.
Right now, the only thing in my way is that the server was restarted last night and the Job Agent wasn't part of the restart....so I'm waiting for server support to get back from lunch so that I can test out my implementation.
If anyone has any questions about my approach, I'd be more than happy to share.
"I need build Skynet. Plz send code"
|
|
|
|
|
Never actually deleting records can provide an extra layer of data protection.
A common approach is to have an IsDeleted flag on each database table to specify whether the record is "deleted" or not. However, without the proper low-level mechanism in place to filter out the records, bugs can begin to invade the application space (and they're not always easy to spot!).
In my case, I have written my own Entity classes to handle CRUD operations. The SELECT methods within these classes automatically filter on IsDeleted and the classes are auto-generated by a tool I wrote to ensure this issue is minimized. However, I recognize some folks may want to use other techniques to address this issue.
What technique(s) do you prefer? My goal is to initiate some dialogue and gain some feedback as to the various mechanisms used by fellow members on this particular subject matter. Thanks in advance.
|
|
|
|
|
The first thing that comes to mind, is the method used in the SAP ERP package that we use: history tables.
These tables are pretty much copies of productive tables, but with a different key structure (don't know exactly how), and are used to track changes to records, and keep copies of deleted records.
So for example, when I delete a product card from the system, it really is deleted from the products table, but a copy is left in the products-history table, with the status 'deleted'.
It works well for straightforward objects such as products and business partners, but there are volatile objects such as sales documents (having header and row data) that can cause the database to grow beyond application specs.
My advice is free, and you may get what you paid for.
|
|
|
|
|
Excellent idea. Simple, yet effective.
Here are some ideas to expand on that idea.
The history table could contain the following fields:
1) identity column which is the primary key
2) Date/timestamp of the user who performed the delete
3) A XML column where you could store an XML representation of the data that was deleted.
This provides a straight forward method to support all kinds of table layouts.
Maybe this logic could be implemeted as a stored procedure and your developers wouldn't even know about these gruesome details.
My 2 cents.
Good luck 
See here about storing XML documents in SQL Server.
http://msdn.microsoft.com/en-us/library/ms345117(v=sql.90).aspx[^]
|
|
|
|
|
I do not see much advantages, unless both tables are located in separate files. Otherwise you'd be adding a lot of extra work, just to give it a new status. I'd still prefer having them in the same table, with an (indexed) attribute to make a distinction between the records with the status "deleted" and the ones that aren't.
Yes, you can omit checking that single attribute in each query on that table, that could be considered an advantage. There's more disadvantages; you'll be maintaining two definitions of the same table in each database, and it'd add extra strain in terms of physically moving the data and updating indexes.
Most of the time, a simple DateTime to indicate the date/time of removal (null if not removed) is enough to provide simple "undelete" functionality. That's from a YAGNI/KISS viewpoint - it might make sense to have history-tables depending on your needs.
In that case I'd recommend to add some triggers and log it in a copy of the database, located on the same server. You can easily write a script that clears all tables in a database, and it'd be easy to make separate backup-plans.
Bastard Programmer from Hell
|
|
|
|
|
Member 8037172 wrote: What technique(s) do you prefer?
My technique is to first start by looking at the business requirements of the application and the enterprise.
Attempting a solution without doing that is pointless.
For example there is a significant difference in user auditing versus supporting an undo.
Not to mention that volume for any situation is a significant factor. If you expect 1000 inserts/updates a second it much more significant than if you expect 1 an hour.
One must at least consider if time should be spent on archiving strategies.
|
|
|
|
|
I'm with JSchell here, it really is a requirements decision rather than a general rule. Different databases serving different types of business needs require different delete strategies.
I often use a disabled flag on the record but never make it a general rule. About the only general rules we have are an audit trigger (spit I hate triggers) and Modified/ModifiedBy on every table.
Never underestimate the power of human stupidity
RAH
|
|
|
|
|
I would create a View with a WHERE IsDeleted = false condition and use the view in my select methods. And I would also make sure that the IsDeleted column is indexed.
|
|
|
|
|
hi all
suppose i have a table named "PlaceHolder" with this fields
ID int
TableName nvarchar
FieldName nvarchar
these fields will indicate specified table.field in my databse
so i want to retrieve them and use it`s content as a tableName.fieldName in my select transaction
SELECT [SQL1].[NewFieldName]------> REPLACED WITH '[TableName].[FieldName]' RETREIEVED BY INNER SELECT FROM
(SELECT '['+[TableName]+']' + TableName + '.['+[FieldName]+']' NewFieldName
FROM [PlaceHolder]
WHERE ....
) AS SQL1
WHERE ...
it will show me the names of some [tables].[field] but i want it`s values in these fields
|
|
|
|
|
Construct a dynamic query in a string and use EXECUTESQL to execute the query.
|
|
|
|
|
I prefer to use String.Format to construct something like that; all those quotes, plusses, and brackets get in the way of understanding what's going on.
Otherwise it should work fine.
|
|
|
|
|
Hi try this
Input:
Declare @tblPlaceholder table(ID int,TableName Nvarchar(20),FieldName Nvarchar(20))
insert into @tblPlaceholder
select 1,'spt_values','name' union all select 1,'spt_values','number' union all
select 2,'spt_monitor','lastrun' union all select 2,'spt_monitor','cpu_busy'
--select * from @tblPlaceholder
Query:
Declare @t table(ID int,Query VARCHAR(2000))
Declare @QueryList VARCHAR(2000)
Declare @i int
set @i = 1
-- Step 1: Build the query and insert the same into a table variable
Insert into @t
Select ID, Query = ' Select ' + FieldNames + ' from ' + TableName
From
(
Select ID,TableName, FieldNames = Stuff((select ',' + CAST(TableName as Nvarchar(20)) + '.' + CAST(FieldName as Nvarchar(20))
from @tblPlaceholder t2 where t2.ID = t1.ID
FOR XML PATH('')),1,1,'')
From @tblPlaceholder t1
Group By t1.ID,t1.TableName)X
-- Step 2 : loop thru the ID and execute the queries
While (@i <= 2) -- since there are two id's.. this can even be configurable as max(id)
Begin
SELECT @QueryList = (Select Query from @t where ID = @i)
exec(@QueryList)
set @i += 1
End
/* Final output
Output of Select spt_values.name,spt_values.number from spt_values
name number
(rpt) -32768
YES OR NO -1
SYSREMOTELOGINS TYPES -1
SYSREMOTELOGINS TYPES (UPDATE) -1
Output of Select spt_monitor.lastrun,spt_monitor.cpu_busy from spt_monitor
lastrun cpu_busy
2008-07-0916:46:13.877 10
*/
I have given the demonstration by using two system tables (spt_values and spt_monitor) found in the Master database.
The comments about the steps are mentioned in the code itself. Hope this will help you
Let me know in case of any concern.
Thanks
Niladri Biswas
|
|
|
|
|
hi
thanx to ur answer
it took a time fore me to understand what just going on
i will try it and answer back
  
|
|
|
|
|
Hi,
When I run the following select getdate(), I'm getting the value below:
2011-07-04 13:54:22.427
I want to get only the value 2011-07-04...
Can anybody advise?
Note: I don't want to convert it to varchar.
Cheers
|
|
|
|
|
|
If you are using SQL Server 2008 then use the date datatype
select cast(getdate() as date)
returns 2011-07-04
Lobster Thermidor aux crevettes with a Mornay sauce, served in a Provençale manner with shallots and aubergines, garnished with truffle pate, brandy and a fried egg on top and Spam - Monty Python Spam Sketch
|
|
|
|
|
|
Without using the convert function to manipulate the date then your best answer is from thatraja
Lobster Thermidor aux crevettes with a Mornay sauce, served in a Provençale manner with shallots and aubergines, garnished with truffle pate, brandy and a fried egg on top and Spam - Monty Python Spam Sketch
|
|
|
|
|
SELECT DATEADD(dd,0,DATEDIFF(dd,0,GETDATE())) does the trick.
|
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
Try this