Asked By – Björn Pollex
I have a multi-line string defined like this:
foo = """ this is a multi-line string. """
This string we used as test-input for a parser I am writing. The parser-function receives a
file-object as input and iterates over it. It does also call the
next() method directly to skip lines, so I really need an iterator as input, not an iterable.
I need an iterator that iterates over the individual lines of that string like a
file-object would over the lines of a text-file. I could of course do it like this:
lineiterator = iter(foo.splitlines())
Is there a more direct way of doing this? In this scenario the string has to traversed once for the splitting, and then again by the parser. It doesn’t matter in my test-case, since the string is very short there, I am just asking out of curiosity. Python has so many useful and efficient built-ins for such stuff, but I could find nothing that suits this need.
Now we will see solution for issue: Iterate over the lines of a string
Here are three possibilities:
foo = """ this is a multi-line string. """ def f1(foo=foo): return iter(foo.splitlines()) def f2(foo=foo): retval = '' for char in foo: retval += char if not char == '\n' else '' if char == '\n': yield retval retval = '' if retval: yield retval def f3(foo=foo): prevnl = -1 while True: nextnl = foo.find('\n', prevnl + 1) if nextnl < 0: break yield foo[prevnl + 1:nextnl] prevnl = nextnl if __name__ == '__main__': for f in f1, f2, f3: print list(f())
Running this as the main script confirms the three functions are equivalent. With
timeit (and a
* 100 for
foo to get substantial strings for more precise measurement):
$ python -mtimeit -s'import asp' 'list(asp.f3())' 1000 loops, best of 3: 370 usec per loop $ python -mtimeit -s'import asp' 'list(asp.f2())' 1000 loops, best of 3: 1.36 msec per loop $ python -mtimeit -s'import asp' 'list(asp.f1())' 10000 loops, best of 3: 61.5 usec per loop
Note we need the
list() call to ensure the iterators are traversed, not just built.
IOW, the naive implementation is so much faster it isn’t even funny: 6 times faster than my attempt with
find calls, which in turn is 4 times faster than a lower-level approach.
Lessons to retain: measurement is always a good thing (but must be accurate); string methods like
splitlines are implemented in very fast ways; putting strings together by programming at a very low level (esp. by loops of
+= of very small pieces) can be quite slow.
Edit: added @Jacob’s proposal, slightly modified to give the same results as the others (trailing blanks on a line are kept), i.e.:
from cStringIO import StringIO def f4(foo=foo): stri = StringIO(foo) while True: nl = stri.readline() if nl != '': yield nl.strip('\n') else: raise StopIteration
$ python -mtimeit -s'import asp' 'list(asp.f4())' 1000 loops, best of 3: 406 usec per loop
not quite as good as the
.find based approach — still, worth keeping in mind because it might be less prone to small off-by-one bugs (any loop where you see occurrences of +1 and -1, like my
f3 above, should automatically trigger off-by-one suspicions — and so should many loops which lack such tweaks and should have them — though I believe my code is also right since I was able to check its output with other functions’).
But the split-based approach still rules.
An aside: possibly better style for
f4 would be:
from cStringIO import StringIO def f4(foo=foo): stri = StringIO(foo) while True: nl = stri.readline() if nl == '': break yield nl.strip('\n')
at least, it’s a bit less verbose. The need to strip trailing
\ns unfortunately prohibits the clearer and faster replacement of the
while loop with
return iter(stri) (the
iter part whereof is redundant in modern versions of Python, I believe since 2.3 or 2.4, but it’s also innocuous). Maybe worth trying, also:
return itertools.imap(lambda s: s.strip('\n'), stri)
or variations thereof — but I’m stopping here since it’s pretty much a theoretical exercise wrt the
strip based, simplest and fastest, one.
This question is answered By – Alex Martelli
This answer is collected from stackoverflow and reviewed by FixPython community admins, is licensed under cc by-sa 2.5 , cc by-sa 3.0 and cc by-sa 4.0