An election app

This election app has seen some rapid evolution since its first iteration spearheaded by kevin schaul a year ago.

Since then we have completely tore it apart and started from the models up.

The idea:
An app that would house data of past and future elections accessable by logical urls.
Each election would have a study guide component, a live results component and an archive component.

The reality
We are still working on the the transitional phases but the three components do exist. An app is never done. If you wait it will never launch.

Tools:
new relic
django
django-tastypie
django-positions

How we did it:
A team of four, Ben Turner(performance,backend), Lauren Rabaino(templates,design), Jon Schleuss(fact checking, front end) and of course me(all of the above) spent about 6 weeks of non-stop development using lessons from previous iterations.

Models:
This app is really three apps. People,politicians and elections. A person can become a politician who can be part of an election. Why would we add this layer of abstraction? People are crazy and do a lot of things. A person could be a ceo, a politician, a candidate, a race car driver and on and on. We structured our models to take this into account in hopes that later, maybe much later, we would have a wealth on data on a person through out their career.

One of the problems we ran into is that different people have different info. Everyone has a name, everyone has a birthday. Not everyone has tattoo on their arm or a pet named “the sargent”. There was no feasible way to include all of the possible fields on the model with out bloating the form. Solution? Custom fields. WordPress knows all about it. These “custom fields” allow us to attach seemingly arbitrary data to any model. Brilliant? Not so fast. These relationships are costly at the database level. Ben was able to use some django magic with select_related()

Sorting anybody? Also a problem. We finally decided on the django-positions app. This app allows us to pick an number for an object. It will then automatically reorder the entire list relative to the new number on save. Quite handy. We can now determine what order races are shown in a county, or what order candidates are shown in a race. Random, alphabetical and numeric are no longer our only choices. Ben is in the process of writing some javascript so we can drag and drop order objects.

Views:
With a short amount of time, not enough though was put into how we would display the data. Our views include: all elections, elections by year,all races in an election, race detail, and candidate detail. We added one view to show races flagged with featured for a given election. In the future we hope to do something a little more interesting. Maps? boundaries? addresses? Iterative people, iterative.

Data:
The data comes from two sources. The washington secretary of state and king county.

I wrote a management command called collect_results that parses each source and puts it into the db using the django orm. This data is dirty so a lot of time was spent writing functions to clean it up. For example the five word name, or the race that was in multiple counties or candidates named Write-in.

Custom races:
Some races attract more attention than others. For example the seattle city mayors race. One of our reporters Emily H. did some amazing data collection through reporting. She was able to enter it into the django admin. There is a custom template argument in the race detail view that allows us to create

API!
One of my favorite things about this project is the tasypie-driven api. We work outside our main cms. We need to get data into it. This api is a great solution. We can build javascript widgets to put on story pages, section pages and even our home page. The data displayed is controlled with parameters in a url. An example of this usage was our homepage widget on elections night. We wanted to display the top candidates and their results for the mayoral race. the url looked like this: elections/2013/primary/mayoral-race
This url contained the json for a single race that could be consumed by some simple js.

This becomes extremely useful for instance if we wanted to show the results of a race on a story written for that race. They are also great dynamic teasers into our app.

Overview:

All of this info is available in many places. We hope to make it easily navigated and be an archive for years to come. It will only get better with each election.

Why I develop in the newsroom

There are so many reasons why a developer would want to work in a newsroom. Here are a few supporting why the newsroom should want a developer to work in the newsroom.

Amazing things can be done when you get a reporter, a designer and a developer together. Each brings ideas, perspective and skills to the table that are needed to present a story on the web.

Our newsroom does not have dedicated developers but my how we try. And this is why.

What doesn’t work:
– Developers on different floors, in different departments with no news room presence.
– A shared group of developers encompassing the entire company’s website needs.
– A convoluted process that interrupts collaboration.
– A rotating developer that sits in the news room for one week at a time

Why it doesn’t work:
There is a layer of communication projects go through when the developer is in a different department. Newsroom goals do not have the same priority and often get scaled down due to other priorities. The development becomes request #324 instead of a project. Request #324 competes with requests #325,#326 and so on.

What does work:
– A team of people collaborating from the beginning to make sure each of their pieces fits seamlessly together.
– Face time with developers even when there is no specific project being worked on.
– The idea that the developer is part of the newsroom

Why it works:
When the developer is able to harness the excitement and enthusiasm of the people coming up with the ideas the possibilities are endless. Real relationships are formed that aid the creative process. The dev is no longer a nameless person working from an obscure dungeon. Reporters and designers start to understand what the developers actually do and it helps planning and ideas.

Here’s to hoping more newsrooms can come to see the importance of dedicated developers even if it means shifting around budgets and shaking up outdated mentalities.

Git deployment with dreamhost

Today I attempted to set up deployment by adding a remote to my repo located on my server hosted by dreamhost. One giant misunderstanding lead to hours of problems.

When you push to a git repo, you are pushing changes, not actual files. In order to achieve the deployment I was looking for you must have use a git hook.

Below are steps to set up git with a hook file I found, somewhat specific to dreamhost.

cd into the root of your repo
git init
git config receive.denyCurrentBranch ignore   
curl https://raw.github.com/gist/2715777/c44c8f05711182c58aed2dace960614729aa916c/git_dreamhost_deploy.sh > .git/hooks/post-update
chmod +x .git/hooks/post-update  

Next I needed to add the remote. It to me a while to realize dreamhost required ssh prefix. Below is the code I ran to add my subdomain as a remote.

git remote add staging ssh://username@domain.com/~/sub.domain.com/.git
git push live master

Back on the server run

git checkout master

to actually see the files.

Makinig a map using mabox, tabletop and google spreadsheets

Mapping people, mapping!
Google has made it a snap to create a map using fusion tables. If you are short on time definitely start there. Here at the Seattle Times we wanted something a little more custom.

As a disclaimer this is not a tutorial but a high level rundown of how to get your data on to a map.

There are many map api options:
googlemaps
leaflet
bing
openlayers.org
and many more

We chose mapbox for great documentation and it’s integration with leaflet. For more advanced maps there is great integration with tilemill Also helps that John Keefe at WNYC has lead the way.

What we got:
We can update a spread sheet and points will display on a fully customized map. How? Check this out.

Start with a google spread sheet and Tabletop
With in a few minutes you have an html page pulling in structured data from your spreadsheet. Nice!

Next we take that data and turn it into geojson, a format used by mapbox’s api to plot points. Geo json is regular json with a geometry and properties object.

{
    type: 'Feature',
 
    properties: {
        title:title,
        'marker-color': '#3887BE',
        description: description,
        category: category,
        favorite:point.favorite(),
        "marker-symbol": markerStyle
    },
        geometry: {
            type: 'Point',
            coordinates: [point.longitude(),point.latitude()]
    }
}

Beautiful. Now what?

Using the mapbox api we initialize a map, define its center, zoom level and what tiles to use.

this.map = L.mapbox.map(element);
this.map.addLayer(L.tileLayer('http://tile.cloudmade.com/e283d06f40b246c0a9a7f7b4ed2eba51/95166/256/{z}/{x}/{y}.png'));
this.map.setView([47, -123], 7);

Next we define a layer of markers. Each marker is defined with geojson structure above. Once this layer is created we add it to our map.

this.markerLayer = L.mapbox.markerLayer(this.markers,{sanitizer: function(x) { return x; }}).addTo(this.map);

Boom, you have a map with points on it. Ok, but we had a few requirements.

  • show pop up content in a static side bar
  • consolidate points with the same coordinates into one
  • filter based on category
  • show different icons for different categories
  • customize tiles
  • control the look and feel of the entire map

Using the api we were able to do all of these things fairly easily.

show pop up content in a static side bar
Before we add the markers layer we change the click functionality to fade in the sidebar and push the popup content into the sidebar.

            this.markerLayer.on('click',function(o) {
                $("#popup-content, .sidebar").fadeIn();
                alert.html(o.layer._popup._content);
            });

consolidate points with the same coordinates into one

To do this we use javascript to reformat the data we got using tabletop from the google spreadsheet.
As each point comes in we check if it exists, if it does we append the popup content from the existing point to the new point. Then remove the old point.

This is a little confusing but in the end we get a single point with the data from all the entries from the same point.

show pop up content for all points in sidebar

For this we use mapboxes setFilter function. This function again loops over the points and removes any that don’t have the category. Nice!

show different icons for different categories

Mapbox has a great custom icon option for their markers. Unfortunately we needed even more customization. There is definetly a better way to do this. With that said this is how I did it. Loop over the markers, check for a category, swap our the src for its marker img with the my custom one. It works.

customize tiles

This is huge and deserves an entire collection of posts dedicated to it. The possibilities are endless. Our team was on a budget so we went with cloudmade. They offer 500 thousand tiles for free. More than enough for our usage of this map. Our designer was able to quickly customize a tile set.

I used this url which includes our api key and our tile id to add it to the map.

this.map.addLayer(L.tileLayer('http://tile.cloudmade.com/<api key>/<tile id>/256/{z}/{x}/{y}.png'));

The final product was a map that we were able to customize.

Sound easy? I had my share of issues that come with learning a new api.

Old API’s:
I was using an old api. Be very aware of this when googling help for mapbox. Most answers and tutorials use an outdated api that is not compatible with the new one. I basically built this map twice with two very different api’s. The good news is the new api is compatible with leaflet’s. This means what you can’t find in mapbox’s documentation you can probably find in leaflets. Very nice!

Mapbox sanitizes popups
I could not get images to show up in the popup content. This is because mapbox runs popup content through a sanitizer stripping out anything that makes an external call, for instance images. Why? Not sure.

fix that disables sanitizer:
L.mapbox.markerLayer(this.markers,{sanitizer: function(x) { return x; }}).addTo(this.map);

Well that is our map in a nutshell. You can find the actual code on github if you want to dig deeper.

Bonus! script for geocoding a spreadsheet:
http://mapbox.com/geo-for-google-docs/

Workflow

A corporate word that constantly “needs to be analyzed” or an actual thing that with the right ingredients and actually make your life better?

Probably both. There seems to always be a trade off. Initial set up for prolonged time savings. I find myself saying, it is faster to make this change this way this time. In the future I should do something different. Some time the future comes with in a week, some times it comes within a year, and sometimes it never comes.

Most bang for your buck.

Codekit:

control-save, control-tab, control-r. Repeat. Sound familiar? I mean these keys must ship with reinforcement or something. It gets crazier if ftp is a part of your workflow.
Codekit compresses that step to control-s. How?
If your editing a css file upon save it injects the new css and animates in the changes. This time saver might be hard for you to justify because you can do all three of those in less than a second. Believe me, it adds up.
Just let codekit take care of it for you. I’m not doing codekit justice by touting this one feature but you can see all the other essential functions it has. (sass, compass, image compression anyone?) Check it out.

A local development area:

Whether your working on a worpress site, django or any other cms you will save your self a ton of time by taking out that dreaded ftp step. This may sound like a given but I know we have all sat down to make a quick change and an hour later we are still control save, control tab, control r. Ftp. Repeating. Just don’t do it.

This is easier to enforce if you can get rid of ftp all together and deploy from a source control like git hub. Highly recommended.

Really learning your text editor:

So many times a day I will think to myself, there must be a command for this. Turns out, there almost always is. I use coda and this a an awesome tip sheet for starters.
Things like:

  • wrapping something in a tag,
  • multiple clipboards for complex copy pasting,
  • selecting and editing multiple columns at once,
  • opening files by typing there names(not drilling down in folders)
  • storing commonly used clips and tons more.

You get the idea. And if you can’t find what your looking for you can write a plugin. Spend time to save time.

Get in good with that terminal:

commit and deploy with the terminal. I’m convinced there is no faster way and it opens up a world that you will inevitably need to get closer with.

More bang for more bucks

And then there are those things that add to your workflow but save you in the long run.

Sass
Git
Frameworks of all sorts, django, angular, mustachejs, and the list goes on.

Each time you add one of these to your workflow document it!!! I am forced to work on many machines developing the same projects. I lose tons of time trying to get each of these workflows working on new machines. Even if you don’t need to do this on the daily there will come a time and you will thank me.

jQuery ajax issues with IE

This is a fix to an obscure issue I ran into. It will most likely fix any of your ie ajax issues.

Ajax with jQuery is unbelievably simple. You will find all you need to know from this tutsplus tutorial.

To load our static app all that’s needed is this code. It makes the request and loads the html into the div with and id of ‘result’. Fantastic.

var loadUrl = 'url_to_app';
$("#result").load(loadUrl);

Per usual I was experiencing an issue with IE.

LOG: Given URL is not allowed by the Application configuration.: One or more of the given URLs is not allowed by the App’s settings. It must match the Website URL or Canvas URL, or the domain must be a subdomain of one of the App’s domains.

WTF is that? Ajax fails silently so I’m not sure this error is even related other than the fact it went away after I used the solution below.

I know jQuery’s .load() does work in ie so my conclusion was the part pushing the result into the #result div was throwing the log message and failing. I found threads noting that broken html could be the cause or headers that ie does not like. The ajax call was being made to s3 and I did not want to mess with their header info. I could not find any issues with my html. Frustration ensued.

The ‘fix’. Not ideal but got the job done.

load();
//funtions for xdr ajax
function alert_error()
{
}
function alert_loaded()
{
$("#result").html(xdr.responseText);
}

function load(){
var url = 'url_to_app'+'?'+new Date().getTime();//stops ie from caching
if ($.browser.msie && window.XDomainRequest)
{
xdr = new XDomainRequest(); // Create a new XDR object.
xdr.open("get", url);
xdr.onerror = alert_error;
xdr.onload = alert_loaded;
xdr.send();
}else{
$.ajaxSetup ({
cache: false
});
var loadUrl = url;
$("#result").load(loadUrl);

}
}

Whats happening here? A quick check to see if the browser is ie and accepts XDomainRequests. If so we use that protocol to make the request. Note the time stamp query string to break any cacheing.
Upon success it pushes the content into our #result div.

If any other browser we get to use the far more condensed .load(). Same outcome.

Adding multiple many to many fields to a Django model object

This is a ramble but,

I recently needed to scrape a website and add a bunch of bills to a politician.

The scraper can be found at https://github.com/deanohyeah/Legislator-Scraper

Once the data has been scraped you add it to each object accordingly. For the bills however you need to run it through a for loop and add each bill individually to the candidate.bills object.


for key,value in billsDict.iteritems():
billNum = str(value['billNumber'])
the_bill, created = SponsoredBill.objects.get_or_create(
bill_number = billNum,
description = 'who are you',
url = 'test',
)
candidate.bills.add(the_bill)

I was running into:
RuntimeError: maximum recursion depth exceeded while getting the str of an object
This was due to beautiful soup giving me a NavigableString vs a standard string. I cast the variables fron m the dict to str() and it solved the problem.

The magic comes from candidat.bills.add(the_bill)
Each bill it is added to the bills object for the candidate.

Woooaaahhh!! NYTimes

I’m sure by now everyone has seen the “first look” of the nytimes redesign. If not go here.

For the longest time I’ve taken solace in the fact that even the New York Times can’t get their act together to create a modern, responsive, easy to navigate site.  Maybe we are not too inept. I see this redesign as a double edged sword. On one hand, it is beautiful, simple and stunning. Drool worthy if you will. On the other, it is a natural progression that begs the question, what took so long and why did they get to do it first?

NYTimes has truly out done themselves and I am jealous!

I think the biggest innovation here will be the article page. Big beautiful photos, a sidebar that makes sense, and a flexible layout. It gives the feel that every article is an experience, not just a written story.

I thank you NYTimes, maybe you will open the eyes of other news organizations. Just because we print a news paper, doesn’t mean our website needs to look like one.

Show me what I want! A catered web experience.

How great woud it be to go to a website and not have to use a nav because it already knows why you are there? Creepy? Very. But also efficient. I’m not going to delve into mind reading but I would like to take advantage of any info a visitor is willing to supply. In the end it can help them find what they are looking for quicker.

Ideas include using geo location, frequency of visits and and the battery api to put together a somewhat personal experience. I want to be responsive not only to screen size but also situation. If you have read my about, you probably don’t need to see that on the home page. If you’ve read this article you don’t need it on the home page. If your battery is running low i’ll slow down on the turbo charged javascript heavy interactive. If you are anywhere near me I will promote my contact info a little more.

Ideas for this concept are unlimited. Lets make it happen people.