Multi tool use

How to check a Python Set for a string

Hopefully this doesn't get downvoted on the title but couldn't think of a better way of explaining the issue.

Based on suggestions I've seen on Stack, I am using Sets to ignore duplicate lines which works a treat until I get a use case where the line changes slightly but I still want to filter that line out as a duplicate. In my example, I cannot seem to be able to search a string for a certain keyword so in my example below, I exclude any new line where the firs column, which is the ID, already exists, in this case London.

For example.

London,Sold,2021-12-07,1000000,301909
London,Sold,2021-12-07,1000000,999999


So I wanted to know if it was possible to just check whether the ID London exists in my Set before adding the line or ignoring but I can't find any way to do this. I tried tuples but not sure that is my solution and I cannot create a set from list. My simple test case is as follows and the end result is I only get the line in my set, not the testline as is happening now.

testline = 'London,Sold,2021-12-07,1000000,301909'
id = 'London'

j="testline2"
seen = set()
seen.add(testline)

if id not in seen:
seen.add(j)

print seen


4 Answers
4

It seems like you want a dict, where the key is the first value, rather than a set.

seen = {}
id = testline.partition(',')[0]
seen[id] = testline
...
if id not in seen:
...


id

testline = 'London,Sold,2021-12-07,1000000,301909'
id = 'London'

j="testline2"
seen = set()
for element in testline.split(',')
seen.add(element)
if id not in seen:
seen.add(j)

print seen


You're near to the solution:

testline = ['London','Sold','2021-12-07','1000000','301909'] #updated
id = 'London'

j="testline2"
seen = set(testline) #updated
if id not in seen:
seen.add(j)

print (seen )


testline

You need to extract the first split of your comma separated string and add it to your set. For this, you can either your x.split(',', 1)[0] or x.partition(',')[0].

x.split(',', 1)[0]

x.partition(',')[0]

The logic is identical to the commonly used itertools recipe unique_everseen, available in the official docs.

itertools

unique_everseen

The recipe has been re-implemented in 3rd party toolz.unique, which can save you reinventing the wheel:

toolz.unique

from toolz import unique

L = ['London,Sold,2021-12-07,1000000,301909',
'London,Sold,2021-12-07,1000000,999999']

res = list(unique(L, key=lambda x: x.partition(',')[0]))

print(res)
['London,Sold,2021-12-07,1000000,301909']


By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.