A Look At Hoodline's Internal Local Data Tool, "Hoodbot"

One of the most important resources for us at Hoodline is publicly-available local data.

Our writers constantly scour information like business registrations, liquor license filings, crime reports and planning applications as they research stories, detect trends, and break news.

Unfortunately, accessing this data is pretty hard.

Even in a tech-savvy city like San Francisco, where much of this data is published daily online, there remains a lot of friction. Different agencies have their own data structures, file formats, and publishing schedules. Data may be strewn across a dozen different websites, and may not be easily searchable (or even searchable at all). In one particularly quirky case, an agency that shall not be named publishes new records only via a daily, manually-generated email.

Checking these datasets for relevant records each day, or searching across them for a specific piece of information, was proving prohibitively time-consuming for our writers and editors. So, we built a solution.

Internally, we call it "Hoodbot." And it looks something like this:

The Hoodbot dashboard

We've hooked into public datasets at the city, county, and state level, allowing our writers to search and sort all relevant data from a single interface. The records are organized by dataset, but can also be searched by keyword, or sorted by neighborhood.

As new data is published, Hoodbot scans it, cleans it up, and (for data points with geographic locations), geotags it and associates it with relevant neighborhoods. So, for example, our Outer Sunset editor can fire up Hoodbot each morning and see new liquor license applications filed only in her neighborhood, or recent SFPD reports in the area.

A bonus feature: Hoodbot is integrated with Slack, the team chat application we use all day every day at Hoodline. New data records are pushed from Hoodbot into a Slack channel where they are automatically tagged by neighborhood and badged by dataset. So our editorial operation gets relevant data pushed to it automatically, as soon as it's available, integrated seamlessly into its existing workflow.

Our Hoodbot data Slack channel

Hoodbot is already proving invaluable for our writers, but we view it as just the beginning. Future enhancements include adding more datasets, enabling additional notifications when new data matches a user's preferences, and even making a public version of Hoodbot available to our readers. Because while innovations like Hoodbot help us streamline and empower our newsroom operation, ultimately our mission is to help our readers better understand the world around them. And if we derive this much value from easy access to local data, we're pretty sure our readers will, too.