how to remove duplicate records in a json file.

Question

4.00/5 (2 votes)

See more:

Hi,
I am reading data from joson file using like this

C#

using (StreamReader r = new StreamReader(filename))
                {
                    string json = r.ReadToEnd();
                    dynamic array = JsonConvert.DeserializeObject(json);
                    foreach (var item in array)
                    {
                        Console.WriteLine("{0}", item.Title);
                        Console.WriteLine(item.Company);
                    }
                }

my json file having data like this

{
		"Title": "Financial Analyst",
		"Company": "I Serve Systems Pvt Ltd",
		"Location": "Ganganagar",
		"Skill": "Gas, Oil, Energy, Financial Analysis, Finance",
		"Description": "The areas of Responsibility: * Work flow planning * Client liaison * Detailed ...",
		"Link": "http://jobsearch.naukri.com/job-listings-Financial-Analyst-I-Serve-Systems-Pvt-Ltd-Ganganagar-1-to-4-years-270914002180"
	}

{
		"Title": "Financial Analyst",
		"Company": "I Serve Systems Pvt Ltd",
		"Location": "Ganganagar",
		"Skill": "Gas, Oil, Energy, Financial Analysis, Finance",
		"Description": "The areas of Responsibility: * Work flow planning * Client liaison * Detailed ...",
		"Link": "http://jobsearch.naukri.com/job-listings-Financial-Analyst-I-Serve-Systems-Pvt-Ltd-Ganganagar-1-to-4-years-270914002180"
	}

Above two records are same, but I want only one record.
I want to more efficient way because I have to pass thousands of files at a time.
Please suggest me.
Thank you.

Posted 31-Oct-14 4:49am

NagaRaju Pesarlanka

Updated 31-Oct-14 4:50am

v2

Add a Solution

Comments

Dominic Burford 31-Oct-14 11:05am

What is the process that creates the JSON in the first place? Surely it makes more sense to fix the data at the source than to try to fix it later on in the process. Fix the process that creates the duplicates in the first place.

BillWoodruff 31-Oct-14 12:56pm

The whole point of using JSON is to enable exchange of "objects:" rather than reading a file containing JSON, why aren't you de-serializing a JSON file into a collection of objects using a Class structure that matches the structure of the data ?

Maciej Los 31-Oct-14 14:40pm

Good point!

Afzaal Ahmad Zeeshan 2-Nov-14 9:09am

Wow! I seriously didn't notice this comment, but my answer ressembles it. See my answer, Solution 2.

NagaRaju Pesarlanka 2-Nov-14 23:22pm

I already used this process. but I got This error.
Bad JSON escape sequence: \S. Path '[147].Description', line 1184, position 247.

2 solutions

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Afzaal Ahmad Zeeshan · Answer 1 · 2014-11-02T01:30:00

Since you're using Json.NET to work with JSON data. It will be better to have a class generated for the data, this will provide a basic template for your data to work on it. JsonConvert would convert the JSON notations into that class object, and will convert the class object into a JSON string notation.

Your data can have an class object template as

C#

public class MyObject {
   public string Title { get; set; }
   public string Company { get; set; }
   public string Location { get; set; }
   public string Skill { get; set; }
   public string Description { get; set; }
   public string Link { get; set; }
}

This is the template of your data, and you can change your code to this one instead.

C#

// convert to MyObject
MyObject array = JsonConvert.DeserializeObject<myobject>(json);
// loop; remember, there is only one object here
// so i don't understand your loop logic
foreach (var item in array)
{
    Console.WriteLine("{0}", item.Title);
    Console.WriteLine(item.Company);
}

Removing Duplicates

Above was just a suggestion for you to use. The main answer is this, once you're having multiple items inside your list. A single item would always be unique, duplication occurs in multiple items. So, first of all convert the data into a list.

C#

// convert a list, add values
List<myobject> array = JsonConvert.DeserializeObject<list><myobject>>(json);
// get the distinct items..
// use the .ToList() to convert it back to a list.
array = array.Distinct().ToList();

This above code will first convert the json into list of objects. Then it will select only distinct items after that it will save that non duplicate list to the actual list. You can add a condition to the Distinct method using a Lambda expression.

You can learn more on Distinct method here, http://msdn.microsoft.com/en-us/library/vstudio/bb920306(v=vs.90).aspx[^]

JacobLux · Answer 2 · 2014-11-02T00:54:00

Solution 1

If you are just parsing the JSON to get at the data, perhaps you could just use a json to csv converter then you can just use the functions in Excel to remove the duplicates.

Posted 2-Nov-14 0:54am

JacobLux