Intricate interleaved iteration

Thursday 30 January 2025

Someone asked recently, “is there any reason to use a generator if I need to store all the values anyway?” As it happens, I did just that in the code for this blog’s sidebar because I found it the most readable way to do it. Maybe it was a good idea, maybe not. Let me show you.

If you look at the sidebar on the left, “More blog” lists tags and years interleaved in an unusual way: two tags, a year, a tag, a year. That pattern of five repeats:

python
coverage
‘25
my code
‘24
math
beginners
‘23
git
‘22
github
testing
‘21
audio
‘20
(etc)

I chose this pattern because it seemed to fill the space nicely and simpler schemes didn’t look as good. But how to implement it in a convenient way?

Generators are a good way to express iteration (a sequence of values) separately from the code that will consume the values. A simplified version of my sidebar code looks something like this:

def gen_sidebar_links():
    # Get list of commonly used tags.
    tags = iter(list_most_common_tags())
    # Get all the years we've published.
    years = iter(list_all_years())

    while True:
        yield next(tags)
        yield next(tags)
        yield next(years)
        yield next(tags)
        yield next(years)

This nicely expresses the “2/1/1/1 forever” idea, except it doesn’t work: when we are done with the years, next(years) will raise a StopIteration exception, and generators aren’t allowed to raise those so we have to deal with it. And, I wanted to fill out the sidebar with some more tags once the years were done, so it’s actually more like this:

def gen_sidebar_links():
    # Get list of commonly used tags.
    tags = iter(list_most_common_tags())
    # Get all the years we've published.
    years = iter(list_all_years())

    try:
        while True:
            yield next(tags)
            yield next(tags)
            yield next(years)
            yield next(tags)
            yield next(years)
    except StopIteration:   # no more years
        pass

    # A few more tags:
    for _ in range(8):
        yield next(tags)

This relates to the original question because I only use this generator once to create a cached list of the sidebar links:

@functools.cache
def sidebar_links():
    return list(gen_sidebar_links)

This is strange: a generator that’s only called once, and is used to make a list. I find the generator the best way to express the idea. Other ways to write the function feel more awkward to me. I could have built a list directly and the function would be more like:

def sidebar_links():
    # ... Get the tags and years ...

    links = []
    try:
        while True:
            links.append(next(tags))
            links.append(next(tags))
            links.append(next(years))
            links.append(next(tags))
            links.append(next(years))
    except StopIteration:   # no more years
        pass

    for _ in range(8):
        links.append(next(tags))

    return links

Now the meat of the function is cluttered with “links.append”, obscuring the pattern, but could be OK. We could be tricky and make a short helper, but that might be too clever:

def sidebar_links():
    # ... Get the tags and years ...

    use = (links := []).append
    try:
        while True:
            use(next(tags))
            use(next(tags))
            use(next(years))
            use(next(tags))
            use(next(years))
    except StopIteration:   # no more years
        pass

    for _ in range(8):
        use(next(tags))

    return links

Probably there’s a way to use the itertools treasure chest to create the interleaved sequence I want, but I haven’t tried too hard to figure it out.

I’m a fan of generators, so I still like the yield approach. I like that it focuses solely on what values should appear in what order without mixing in what to do with those values. Your taste may differ.

Comments

[gravatar]

I gave a shot at tossing stuff together with itertools.

I got to filter(None, itertools.chain.from_iterable(itertools.zip_longest(tags, tags, years, tags, years))), and I don’t think I want to try any harder.

[gravatar]

I also sometimes prefer to write a generator for expressiveness, but find it more convenient to have a function to return a list. I’ve started writing a small decorator called @collect that converts my generator into a function that returns a list.

Sometimes I think it would be nice to have it available in the standard library (in itertools, or maybe functools?), and I also wonder if other people use something like that and what do they name it.

[gravatar]

@Marius: it would be super handy to have such decorator in the standard library. I would use it a lot.

Add a comment:

Ignore this:
Leave this empty:
Name is required. Either email or web are required. Email won't be displayed and I won't spam you. Your web site won't be indexed by search engines.
Don't put anything here:
Leave this empty:
Comment text is Markdown.