Multi tool use

MemorError while calculating silhouette_score

I am running a KMeans clustering algorithm on a matrix with shape (190868,35). I am running the following code for the same:

for n_clusters in range(3,10):
kmeans = KMeans(init='k-means++',n_clusters=n_clusters,n_init=30)
kmeans.fit(matrix)
clusters = kmeans.predict(matrix)
silhouette_avg=silhouette_score(matrix,clusters)
print("For n_clusters =",n_clusters,"The avg silhouette_score is :",silhouette_avg)


and I am having the following error

Traceback (most recent call last):

File "<ipython-input-6-be918e90030a>", line 5, in <module>
silhouette_avg=silhouette_score(matrix,clusters)

File "C:UsersarindamAnaconda3libsite-packagessklearnmetricsclusterunsupervised.py", line 101, in silhouette_score
return np.mean(silhouette_samples(X, labels, metric=metric, **kwds))

File "C:UsersarindamAnaconda3libsite-packagessklearnmetricsclusterunsupervised.py", line 169, in silhouette_samples
distances = pairwise_distances(X, metric=metric, **kwds)

File "C:UsersarindamAnaconda3libsite-packagessklearnmetricspairwise.py", line 1247, in pairwise_distances
return _parallel_pairwise(X, Y, func, n_jobs, **kwds)

File "C:UsersarindamAnaconda3libsite-packagessklearnmetricspairwise.py", line 1090, in _parallel_pairwise
return func(X, Y, **kwds)

File "C:UsersarindamAnaconda3libsite-packagessklearnmetricspairwise.py", line 246, in euclidean_distances
distances = safe_sparse_dot(X, Y.T, dense_output=True)

File "C:UsersarindamAnaconda3libsite-packagessklearnutilsextmath.py", line 140, in safe_sparse_dot
return np.dot(a, b)

MemoryError


If anyone knows any solution to this please suggest. I have tried specifying sample_size = 70000, the code runs and consumes all the memory and the system freezes. I am having a Lenovo Thinkpad with 16GB RAM and a i7 processor.

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.