How to search and compare huge lists of mp3 data fast?

Question

0.00/5 (No votes)

See more:

My prog is a streaming mp3 player. At program start, a file is downloaded containing a list of tracks available online at the moment. Currently there are 20.000.000 items in this list!

This textfile is read and converted to a list(of mediadata), where mediadata is a custom class containg 14 items per track (e.g. artist as string, title as string, album as string, duration as string...)
So I have a list(of mediadata) containing approx. 20.000.000 items.

Locally I likewise have playlists containing roughly 1000 tracks each, existing of the same type (mediadata)

My problem:
Because not all songs online are always available I need to compare these playlists based on "artist" and "title" (the other values may be different!) against MusicLibrary, and if a match is found, I load the data from the MusicLibray into my player.

I can do this with a for each loop of course,but this takes ages to load.

What would be the fastest way to make this comparison? A binarysearch is not possible as far as I know?

Am I approaching this problem the wrong way? Any refreshing ideas?

Posted 22-Feb-15 9:13am

fanoftheplanet

Updated 22-Feb-15 9:16am

v3

Add a Solution

Comments

Maciej Los 22-Feb-15 15:19pm

I'd suggest to use database such as SQL server, MySQL, etc instead of textfile...

fanoftheplanet 22-Feb-15 16:05pm

That would be an idea. The problem with this, is that the program should work offline as well (when there is no access to the database). What I do atm is compare that downloaded list to files that are locally available.

Dave Kreskowiak 22-Feb-15 15:29pm

So you're telling us you want to make 20,000,000 comparisons on data that doesn't have an index? Am I hearing that correctly?

fanoftheplanet 22-Feb-15 16:01pm

A list is indexed, isn't it?
What would be the proper way to do this then?

Dave Kreskowiak 22-Feb-15 22:27pm

Yes, it is, but your file is not. The index has to be built by the List<t> class as you load the data from the file.

The bottom line is storing 20,000,000 records in a file is not the best way to do it. A database is far better suited for this.

Andreas Gieriet 22-Feb-15 16:08pm

How many playlists do you have? And how many entries per playlist? Straightforward: use a database with indexing the 20'000'000 entries.
Or: I guess you have only a few playlist with 1000 entries each, say 20 playlists. If so, 20,000 : 20,000,000 = 1 : 1000 entries. Create a dictionary of the playlist entries by calculating a hash-key based on the relevant attributes (XOR hash code from artists and title, maybe you normalize to lower case first). Instead of loading all 20,000,000 objects into some memory, calculate the same hash key while processing these 20,000,000 items and check if the object is in the (much smaller) playlist dictionary. Maybe worth trying.
Cheers
Andi

fanoftheplanet 22-Feb-15 16:54pm

Thanks Andreas, it's indeed a huge file to be loaded into memory. But I need the additional data for other searches (single tracks a user searches by artist, album, year, genre...)
I was thinking about cutting the size down by grouping objects:

artist-----album----track
........|.........|-track
........|................
........|---album----track
.................. |-track

This would significantly cut down the memorysize and searchtime I guess, but to be honest I have no clue of what collection-type would be appropriate for this kind of structure and would be searchable this way.

Andreas Gieriet 22-Feb-15 17:19pm

If it's more than cross checking playlists, then use the right tools: fill the data into some database and let the database server run the queries. That's what they are build for. You may now re-invent the wheel or learn using the right tools.
Cheers
Andi

Kuthuparakkal 22-Feb-15 16:21pm

var firstNotSecond = list1.Except(list2).ToList();
var secondNotFirst = list2.Except(list1).ToList();

fanoftheplanet 22-Feb-15 16:38pm

...finds only exact matches.

Kuthuparakkal 22-Feb-15 17:00pm

Posted a solution, try it

2 solutions

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Kuthuparakkal · Answer 1 · 2015-02-22T11:00:00

Solution 1

Try this.

C#

var MyListC = MyListA.Where(t2 => MyListB.Count(t1 => t2.Author == t1.Author && t2.Title == t1.Title) == 0);

Posted 22-Feb-15 11:00am

Kuthuparakkal

Frankie-C · Answer 2 · 2015-02-22T11:39:00

Solution 2

Use SQLite locally to store playlist and make exchanges using common SQL commands that can work for local SQLite and the main network database.
This should avoid to reinvent the wheel...

Posted 22-Feb-15 11:39am

Frankie-C