One of the things I needed for my new home page design was a way to split a chunk of HTML to get just the text of the first sentence, which I use for the blog posts on the front page.

The preliminaries: these are Django filters, but mostly they’re just string functions, wrapped with Django decorators to make them useful in Django templates.

Here are two helpers:

@register.filter()
@stringfilter
def inner_html(value):
    """ Strip off the outer tag of the HTML passed in.
    """
    if value.startswith('<'):
        value = value.split('>', 1)[1].rsplit('<', 1)[0]
    return value

@register.filter()
@stringfilter
def first_par(value):
    """ Take just the first paragraph of the HTML passed in.
    """
    return value.split("</p>")[0] + "</p>"

These functions are pretty simple, meant to operate on simple inputs. For example, first_par assumes that the opening tag of the HTML is <p>.

Splitting sentences is fairly tricky. I tried searching for a Python snippet, which I didn’t find. I tried thinking about regexes that could do it, but the rules are too complicated. In the end, the code structure I could understand was to break the text into words, and then add words one at a time to a potential sentence, checking it for sentence-hood.

Here’s the rules I came up with for something being a sentence:

• The end of the sentence must be punctuation (.!?), possibly with closing parens and/or double-quote after it.
• The next chunk of text has to start with an upper-case letter or number, possibly with an opening paren and/or double-quote preceding it.
• The sentence can’t end with “Mr.” or titles like it, or an initial. This is to keep the previous two rules from splitting sentences like “Hello Mr. John Q. Public!” incorrectly in the middle.
• The sentence needs to have balanced parens and double-quotes. This ensures that sentences breaks won’t be identified in quoted material (or parenthetical asides).

These rules seem to work well for picking out the first sentence from each of my 1800-odd blog posts. Here’s the code:

@register.filter()
@stringfilter
def first_sentence(value):
    """ Take just the first sentence of the HTML passed in.
    """
    value = inner_html(first_par(value))
    words = value.split()
    # Collect words until the result is a sentence.
    sentence = ""
    while words:
        if sentence:
            sentence += " "
        sentence += words.pop(0)
        if not re.search(r'[.?!][)"]*$', sentence):
            # End of sentence doesn't end with punctuation.
            continue
        if words and not re.search(r'^[("]*[A-Z0-9]', words[0]):
            # Next sentence has to start with upper case.
            continue
        if re.search(r'(Mr\.|Mrs\.|Ms\.|Dr\.| [A-Z]\.)$', sentence):
            # If the "sentence" ends with a title or initial, then it probably
            # isn't the end of the sentence.
            continue
        if sentence.count('(') != sentence.count(')'):
            # A sentence has to have balanced parens.
            continue
        if sentence.count('"') % 2:
            # A sentence has to have an even number of quotes.
            continue
        break
    
    return sentence

This is coded not for speed but for being able to see what it does and add new clauses as I find broken sentences. The candidate sentence starts out empty. Words are appended to it one at a time, and the sentence checked against the rules. If any rule is violated, we continue to the next word. If all the rules pass, we break out of the loop and return the found sentence.

I know this code isn’t perfect. Here are some things it doesn’t do well:

• Sentences with single-quote quotes, because just counting them isn’t sufficient. Apostrophes and single-quotes make it so that the number isn’t always even.
• Text with curly quotes.
• Sesame Street sentences: “This blog brought to you by the letter B.”
• Sentences about punctuation, or with code in them.

Actually, there are lots of cases that will not be handled well. Word-play enthusiasts I’m sure will enjoy coming up with examples.