Python performance is a fool's errand

I'm beginning to feel like striving for performance in a Python library is counter-productive. Every thing you do to eek out performance warps the code, making it less clear. And it seems like every time you try to make something faster, it actually gets slower.

The issue that's currently chafing my butt is the performance cost of attributes, as it pertains to generators vs. custom iterators. Consider the following two ways to implement a counter iterator...

# Method returning a generator
class A(object):
  def __iter__(self):
    counter = 0
    while True:
      counter += 1
      yield counter
#
# Custom iterator class
class B(object):
  def __init__(self):
    self.counter = 0
  def __iter__(self):
    return self
  def next(self):
    self.counter += 1
    return self.counter

Because it uses local variables, the generator will is roughly ten times faster than the custom iterator.

Currently, when reading through posting lists (the list of documents a certain word appears in), I use simple generators that yield every document in the posting list. I wanted to implement a useful and long-known optimization where you skip through the posting lists to avoid reading document information you don't need. For example, if you're doing an AND between two terms, and the first document in which term A appears is document 10, then in the posting list of term B, you can skip any documents lower than 10.

The problem is that to do this I need to switch from generators to a custom "cursor" type iterator with a method to allow skipping through postings lists. By analogy with the earlier code examples:

class B2(object):
  def __init__(self):
    self.counter = 0
  def __iter__(self):
    return self
  def next(self):
    self.counter += 1
    return self.counter
  def skip_to(self, n):
    self.counter = n

But since this involves a attribute getting/setting and explicit looping, in order to make reading through posting lists faster and more efficient, I first have to make it ten times slower.

ARG!!!

Now, I could try to simulate "skip to" functionality in a generator using a "yield expression" and send(), but

• send() is not supported in Python 2.4 (or earlier).

• I find that sending information into a generator and receiving it from a generator at the same time is counter-intuitive and leads to hard-to-follow code.

• I'd have to call next() and send() in a Python loop instead of using for...in, so the loop would still be roughly four times slower than a straight generator. I'm not sure the speed benefits of skipping through the posting list would make up for the much slower looping.

Another thing makes me wonder about whether trying for performance is worthwhile: hopefully Unladen Swallow will come along some day and reduce or eliminate a lot of performance bottlenecks in the current CPython. It's possible that by crafting my code in weird ways to find the most efficient code paths in the current CPython, I'm writing something that will be slower on a future CPython than a sane implementation.