Quick hack CSV review tool

Tuesday 4 December 2018

Let’s say you are running a conference, and let’s say your Call for Proposals is open, and is saving people’s talk ideas into a spreadsheet.

I am in this situation. Reviewing those proposals is a pain, because there are large paragraphs of text, and spreadsheets are a terrible way to read them. I did the typical software engineer thing: I spent an hour writing a tool to make reading them easier.

The result is csv_review.py. It’s a terminal program that reads a CSV file (the exported proposals). It displays a row at a time on the screen, wrapping text as needed. It has commands for moving around the rows. It collects comments into a second CSV file. That’s it.

There are probably already better ways to do this. Everyone knows that to get answers from the internet, you don’t ask questions, instead you present wrong answers. More people will correct you than will help you. So this tool is my wrong answer to how to review CFP proposals. Correct me!

Comments

[gravatar]
Jonah Bishop 1:18 PM on 4 Dec 2018

Any time I see "spreadsheet" in a case like this (and I see it weekly at my workplace), I think to myself "Why not use a database?" Spreadsheets, in my tiny opinion, are almost always the wrong answer to any problem.

If you're running the conference, presumably you're in control of the website (though I could imagine that not being the case). Proposals should simply be stored in a database table that you then could step through with ease.

[gravatar]
Ben DeLillo 5:09 PM on 4 Dec 2018

Going from what @Jonah Bishop said, MySQL supports CSV directly. Just create a table in the CSV storage engine and replace the empty file it creates with your one with your data https://dev.mysql.com/doc/refman/8.0/en/csv-storage-engine.html

[gravatar]
Ned Batchelder 5:56 PM on 4 Dec 2018

I appreciate the thoughts, but a database doesn't solve the problem I have: how to read multi-paragraph submissions. And it introduces a new problem, which is, how do people get their proposals into the database? Right now we have a Google form that populates the spreadsheet.

[gravatar]
Victor 12:50 AM on 5 Dec 2018

Are there example files available we can use to play around

[gravatar]
Victor 4:56 PM on 6 Dec 2018

Never mind, Initially i thought the csv file needed to have a special schema for this to work.

[gravatar]
Nicholas Chammas 7:21 PM on 9 Dec 2018

Doesn't Google Forms offer multiple views of submissions? One is the spreadsheet view, but another is the individual responses view, where you can see the full responses roughly as they looked like when they were submitted to the form, individual by individual. So paragraph breaks are preserved, and the responses should be easy to read.

[gravatar]
Curt J Sampson 3:36 AM on 27 Dec 2018

@Jonah, a CSV file is a database.

What I think you're actually thinking is, "why not use a DBMS (a database management system, a program that reads, writes and updates databases)?" A DBMS like MySQL or PostgreSQL seems like overkill for this application (though of course if you're already using one for other related data, it's reasonable to take advantage of it for this, too).

In this situation, there is already one piece of a database management system in place: Google Forms, which adds data to a database in a CSV file. CSV is a nice portable format, with lots of easy ways to read, display and write it (Excel and the Python csv module being just two of many) and thus makes it easy to write another part of a DBMS, a reporting application, which is exactly what Ned has done here. And I think it's a great solution. It also follows the Unix spirit of making small tools that work together on common data formats.

For what it's worth, about ten or twelve years ago I wrote a limited RDBMS in Ruby that used TSV files (those being slightly easier to edit in Vim) for table storage and did joins and all the other usual stuff. Given that we were talking truly tiny databases (generally no more than a few thousand rows in a table), performance was just fine. I built a small accounting system on top of this. It was only a few hundred lines of code, development time was little different from trying to build something on PostgreSQL or whatever, and system administration and maintenance were an order of magnitude cheaper. (No servers beyond the Git repo server we already had, backup and a full audit log for free because the TSV files were in Git, and so on.)

Trying to use what some would consider a "proper" system for something that can be done much more simply is one of the biggest time-wasters in software development. It's particularly egregious when the program is used by developers or people who work next to developers; doing a bit of support for these things is far cheaper than trying to write something robust enough to work with less support.

[gravatar]
Giac 3:45 AM on 27 Dec 2018

The real question then is “how do i get data from Google Forms to a nice output”, right? They have an API that would probably allow you to skip the CSV/Excel step and just print out a file formatted the way you prefer: https://developers.google.com/apps-script/reference/forms/form-response

Add a comment:

Ignore this:
Leave this empty:
Name is required. Either email or web are required. Email won't be displayed and I won't spam you. Your web site won't be indexed by search engines.
Don't put anything here:
Leave this empty:
URLs auto-link and some tags are allowed: <a><b><i><p><br><pre>.