Removing duplicate records from SQL database keeping the oldest record

Question

0.00/5 (No votes)

See more:

Hi there,

I have setup a C# application which saves data to a SQL database. I have a button in the application that checks the database for any duplicate records then removes the latest record, by checking the datetime stamp that is added to each record when they are saves to the SQL database.
I now have a great number of records within the database and would like to modify the DELETE statement, which as mentioned previously: this checks all records in the database and removes the duplicate record with the latest datetime stamp.
I now need to modify this statement to only check the last 6 months of records and to ignore if there is any duplicates older than six months.
I have the following SELECT statement which does return the values that I would like the DELETE statement to focus on:

SQL

WHERE createdDate >= (DATEADD(mm,-6,GETDATE())) AND createdDate <= GETDATE()

This is the previous statement that I had setup for removing duplicates from the database, but this checks all records in the database, I need to include a date range into this statement.

SQL

DELETE FROM tblAddress
WHERE A_Id NOT IN (SELECT MIN(A_Id) FROM tblAddress GROUP BY colAddress)
SELECT * FROM tblAddress ORDER BY createdDate ASC

The SQL database includes the following Columns for each record saved to the database:
A_Id = Auto generated ID number, int, PK
colAddress = varChar(255) used to store address
createdDate = datetime, used to track when the record was added to the database.

I need to include this date range into the pre-existing statement under the WHERE clause for removing records, if there is a duplicate record that is older, but within the 6 month range from now:
Current statement to be altered:

SQL

DELETE FROM tblAddress 
WHERE A_Id NOT IN (SELECT MIN(A_Id) FROM tblAddress GROUP BY colAddress)

date range that needs to be included in the WHERE clause:

SQL

WHERE createdDate >= (DATEADD(mm,-6,GETDATE())) AND createdDate <= GETDATE()

Posted 4-Oct-12 15:50pm

saas207

Updated 4-Oct-12 17:30pm

v4

Add a Solution

Comments

Rockstar_ 5-Oct-12 0:51am

For this problem yo have to use the concept of Partition...

4 solutions

Solution 1

Find the pseudo code - Add your own date time difefrence logic

SQL

DELETE FROM Table WHERE A_Id NOT IN
( 
SELECT MAX(A_Id) FROM table WHERE createdDate > 6 months
GROUP BY Id 

)

Posted 4-Oct-12 16:10pm

anshudutta

Updated 4-Oct-12 20:59pm

v3

Solution 2

Hi,

Refer this,

http://www.simple-talk.com/sql/t-sql-programming/removing-duplicates-from-a-table-in-sql-server/[^]

Posted 4-Oct-12 19:23pm

Rockstar_

Solution 3

Hi,

Use the below query.

SQL

DELETE FROM tblAddress WHERE A_Id NOT IN (SELECT MIN(A_Id) FROM tblAddress GROUP BY colAddress) AND createdDate >= (DATEADD(mm,-6,GETDATE());

Hope this will work.

Then check the result again
SELECT * FROM tblAddress ORDER BY createdDate

Posted 4-Oct-12 19:35pm

Mohd. Mukhtar

Updated 4-Oct-12 19:36pm

v2

Comments

[no name] 5-Oct-12 3:01am

Hi Mohd,
I have composed the following statement which includes both of the conditions I require.

DELETE FROM tblAddress WHERE A_Id NOT IN (SELECT MIN(A_Id) FROM tblAddress GROUP BY colAddress) AND createdDate >= (DATEADD(mm,-6,GETDATE()))

When testing this it does select the correct range (within 6 months of now), but the actual DELETE statement will still check for entries earlier than 6 months for a duplicate, instead of just checking for duplicates within the records for the last 6 months.
I do not want to check for duplicate records that are older than 6 months, only want to check for duplicates within the last 6 months.

Mohd. Mukhtar 5-Oct-12 4:05am

See the below ans. That might help you.

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Mohd. Mukhtar · Accepted Answer · 2012-10-04T22:02:00

Ok
Just interchange the condition.

SQL

DELETE FROM tblAddress WHERE createdDate >= (DATEADD(mm,-6,GETDATE())) AND A_Id NOT IN (SELECT MIN(A_Id) FROM tblAddress GROUP BY colAddress);

As the SQL statement execute from left to right, Hence it will check createdDate condition first and then execute other entries, But any how you don't need to check the earlier entries, So the other option is check date condition 2 time as below query.

SQL

DELETE FROM tblAddress WHERE createdDate >= (DATEADD(mm,-6,GETDATE()) AND A_Id NOT IN (SELECT MIN(A_Id) FROM tblAddress WHERE createdDate >= (DATEADD(mm,-6,GETDATE())) GROUP BY colAddress);