How to compute tf-IDF using top 100 occurrences in vocabluary?

See more:

Python

CSV

data

analysis

, +

I'm using a data represented as sparse matrix where column contains numerical values for "article_Id" "word_id" and "count" as follow:

article_Id	word_id	count
0	1	3	1
1	1	10	1
2	1	12	8
3	1	17	1
4	1	23	8

I'm representing each document by TF-IDF vectors of the top 100 coordinates. Next, I'm randomly selecting a subset of 200 documents and computing the similarity of those 200 documents.

Next, the task is to store the tf-idf in a 1000 x 100 dimensional matrix. Here, 1000 is the number of documents and 100 is the top 100 most occurring words. These words will be different for each document, so we have to select top 100 words from whole vocabulary (all documents combined) and compute the tf-idf of these words only for each document.

Since tf-idf is calculated separately for documents, I'm unable to get it for top 100 occurrences from whole vocabulary. Any idea?

What I have tried:

Here's what my code looks like right now.

word_ids = []
article_ids = []
data = {}

document_word_counters = [0] * 1000
word_articles = {}

with open("C:/Users/Mehreen/Desktop/Dataset/20 newsgroups data/data50.csv","r") as f:
     for line in f.readlines():
        line_arr = line.split(',')
        word_ids.append(int(line_arr[1]))
        
        if line_arr[1] not in word_articles:
            word_articles[line_arr[1]] = 1
        else:
            word_articles [line_arr[1]] += 1
        
        if line_arr[0] not in data:
            article_ids.append(int(line_arr[0]))
            data[line_arr[0]] = {}
            document_word_counters[int(line_arr[0]) - 1] = int(line_arr[2])
            
        article = data[line_arr[0]]
        document_word_counters[int(line_arr[0]) - 1] += int(line_arr[2])
        article[line_arr[1]] = int(line_arr[2])

Here's what I'm doing to compute tf-idf.

word_ids = np.unique(word_ids).tolist()
document_vectors = []

for article in article_ids:
    if str(article) in data:
        document_vector = []
        article_data = data[str(article)]
        document_vector = [0] * len(word_ids)
        for key, value in article_data.items():
            index = word_ids.index(int(key)) 
            tf = value/ document_word_counters[article-1]
            idf = 1000 / word_articles[key]
#             print (' so its tf becomes %f and idf %f' %(tf,idf))
            document_vector[index] = tf * idf
        document_vectors.append(np.sort(document_vector)[::-1])            
             
tf_idf_matrices = []

for k in range(100):
    document_count = 0
    tf_idf_matrix = [[0] * len(word_ids)] * 1000
    for document_vector in document_vectors:
        tf_idf_matrix[document_count] = document_vector[:k]
        document_count+=1
    tf_idf_matrices.append(tf_idf_matrix)

and later random sample can be easily chosen with random.sample() function.