Showing posts with label Source. Show all posts
Showing posts with label Source. Show all posts

Monday, November 28, 2016

Event Roundup, Nov 28

By Erika Owens

Event Roundup, Nov 28

You have till Thursday to apply to the JSK fellowships.

Deadlines

Know of any upcoming fellowship or conference proposal deadlines? Have an upcoming event? Let us know: source@mozillafoundation.org.



Read Full Story from Source https://source.opennews.org/articles/event-roundup-nov-28/
This article by Erika Owens originally appeared on source.opennews.org on November 28, 2016 at 08:53PM

Tuesday, November 22, 2016

Low-Budget Natural Language Processing

By Dan Zajdband

Low-Budget Natural Language Processing

Let's talk about natural language processing.

If you’ve ever talked to Siri or performed a Google Search using colloquial language and gotten the right answer, you probably had that magical feeling of being understood by a machine. The discipline that studies the interactions between human languages and computers is called natural language processing (NLP), and it’s a very active field. Companies and computer scientists are developing amazing techniques for improving performance on this task, but adding these features to our sites and apps can be very complex. Even great, free resources aren’t useful if you don’t have the time or skills to use them.

The good news is that we can take advantage of our human ability to analyze natural language and use really simple techniques to assist and amaze our users. I’ll explore a couple of ways to use these techniques in your own projects. These examples use web technologies but can be translated to other platforms and systems easily.

Basic Example: Predicting a Form Field

One of the goals of the Coral Project team while building Ask, a web product that enables news organizations to ask questions of their readers, was to build the form generation side of the project as an API.

One of the benefits of an API is that it allows developers to create their own integrations and user interfaces for creating and editing the forms. To showcase some possibilities, I built an alternative form creator targeting journalists and news devs who were setting up Ask for the first time.

Using NLP techniques to shape UI

When creating a form, it’s important to try and select the most appropriate UI input for the question. This helps the user understand how to complete the information, and it helps us understand the data. Since every question in the questionnaire needs a title, I thought it was the perfect scenario for applying a silly but effective NLP technique. The idea is simple:

  • The user types the question title

  • An algorithm analyzes the content and guess the question type

  • If the algorithm identifies the question type, it returns it—otherwise it returns the default

  • If the user already selected a question type, we consider it made the right choice and do nothing

  • Otherwise we switch the question type select input to the guessed type

I used Preact for writing this website (source code), just because I like over-engineering my experiments. But we can implement this easily with jQuery:

$('.question-title').on('keypress', function() {
  var inferredType = inferQuestionType($(this).val());
  if (!$('.question-type').data('dirty')) {
    $('.question-type').val(inferredType);
  }
});

And of course, this is easy to implement because we didn’t show the hardest part: the inferQuestionType algorithm.

If you want to get really advanced, try this: Before taking a look at this finished inferQuestionType algorithm below, start creating a form yourself. Go to the first question of the form you’re working on, and see if you can figure out what the algorithm might look like. Once you’ve given that a try, check out how I implemented it:

function inferQuestionType (str) {
  if (/e-?mail/gi.test(str)) {
    return 'EmailField'
  } else if (/age|how many|how old/gi.test(str)) {
    return 'NumberField'
  } else if (/^when/gi.test(str)) {
    return 'DateField'
  } else if (/^which/gi.test(str)) {
    return 'MultipleChoice'
  } else if (/\?.*\?/gi.test(str)) {
    return 'TextArea'
  } else if (/phone/gi.test(str)) {
    return 'PhoneNumber'
  } else {
    return 'TextField'
  }
}

That’s it. That’s the way I modeled the English language for my use case. Even if you don’t know what a regular expression is, you can get an idea of how to implement your own model. In case you didn’t try it for yourself or my algorithm didn’t work for you, here is an animated GIF of what you should see:

Screen shot

Is this algorithm covering every possibility? No. Is this going to work in every case? No. But this function runs in microseconds in our user’s browser (I actually measured it); it’s really simple to implement; and it helps most of the users choose the right question type, saving time on form creation.

Testing and improving your model

Once you have your script working, you may want to know the “success rate” which in this case can be something like: “What’s the percentage of cases where the model chose a different question type than the default, and the user didn’t change it?”

This is easy to measure:

  • When we infer the type, we set a variable for the question with the inferred type.

  • On form submission, we check whether the inferred type exists and is different than the submitted type. If so, we lose 1 point. If the types are the same, we win.

How can you store all of these events? An easy way: if you are using an analytics solution, you probably get events for free. I usually send this event to Google Analytics where it’s easy to add the results and get the success rate. After all, this success rate is a measure of the behavior of your users on the site.

Improving your model

You can always improve your model by adding, modifying or removing rules. The good thing is that if our rules don’t detect the user input, you just didn’t help your user, and it will be a normal form—but the app still works as intended. The only thing that can really hurt are the false positives. If we think that every time the user writes “the” the question is (for example) an email, the user will probably need to change the question type every time. That’s a bad scenario. Detecting an option is cool, but good defaults can help a lot.

A More Complex Example: Creating VR from a Text Editor

As part of my OpenNews fellowship, I’m exploring easy ways to produce virtual reality (VR) content. Since VR is a great resource for telling stories, I think it should be easy for people who don’t have coding skills to create prototypes or even full experiences. That belief led me to build GuriVR, an online editor that can interpret natural language. Users describe what their VR scene should look like, in their own words, and Guri VR tries to convert it into an actual VR scene.

Screen shot

Creating something like GuriVR may seem like an impossible mission, especially for someone like me, a developer with no graphics knowledge and who is certainly not a linguist. A lot of people ask me what machine learning techniques that GuriVR uses.

But the domain (for basic VR scenes) is not that broad. We just want to place objects onto the screen in a specific position and probably transition from one scene to another.The idea behind the GuriVR parser is that it’s just one JavaScript file with less than 350 lines of code that runs both in the browser and Node.js, in microseconds—for any story ever created in the editor. That kind of speed is a bonus that anyone can appreciate, whether or not they know how to code. (And it’s especially cool given that the code is a mess, and I totally need to refactor it.)

Creating GuriVR: identifying objects

So how did I make GuriVR work using NLP? It’s all about anticipating user behavior. Here are some names of the objects that GuriVR is designed to support:

  • Text

  • Image

  • Video

  • Panorama

  • Audio

  • 3D Model

This is the basic stuff that anyone wants in a barebones VR scene.

I also got a little fancier and added some aliases for the objects. Some of these are text aliases, like “sound” for “audio.” Users can also input any of these emoji and GuriVR understands them.

  • Image -> Picture Picture emoji

  • Audio -> Sound, Sound emoji

  • Chart -> Chart emoji

You might think adding support for emoji is idiotic, and maybe you are right, but it’s a low-hanging fruit. Emoji, unlike words, are language agnostic, so they are ideal for a lot of scenarios (and really fun to use). Cat

Adding modifiers

For each object that GuriVR supports, I needed to get different kinds of information. For example, a panorama needs a URL, but the text object needs the text content and a position. In some cases when the patterns can be easily identified, I try not to force a format. For example, for URLS, I use a URL regular expression, and for specifying position I use a list of possible values. Listing values is not the best idea for capturing every way that people will try to say something, but for things like position it works most of the time. And it gives you that magical sensation of being understood by the computer. I’m also working on a full 3D positional interface for GuriVR.

(Note: I am working on a full 3D positional interface for GuriVR. Now you can only say: “I have text to my left,” but in the future you will be able to say: “I have text 20 meters to my left and 30 meters above me.”)

When it comes to free-text properties, like the content of a text object, I use a specific format that maybe doesn’t feel very natural. But it does work. I use double quotes (") to enclose text content because it’s easy to parse and feels better than other patterns like {} or [].

To help users actually follow these formats, it’s important to show some examples. In GuriVR, the editor is always pre-filled with a basic scene that holds the language I’m using, to give the user an idea of its patterns. It also highlights the keywords recognized by the parser.

(There is a GuriVR user guide, but there’s a rule that says, “A user interface is like a joke. If you have to explain it, it’s not that good.” Adding explanatory text in the editor isn’t the most elegant solution, but reading a guide? No thanks.)

So that’s it. The “natural language parser” that runs on top of GuriVR is just that one small file, which lists a bunch of words and orders them. Nothing fancy, huh?

Adding moar languages

Screen shot

In case you didn’t notice, GuriVR not only works in English, it also interprets Spanish. What kind of sorcery is this? Well, since I’m just identifying arbitrary keywords and patterns, this kind of silly model works for any language, so just like I added the word “sound” as an alias of “audio” I also added “sonido”—and now it recognizes audio in Spanish. There is no magic around it. Could I add Italian or German? Maybe I will or hey, maybe you send a pull request with the additions.

As Evelyn Eastmond explains in her post about her experience with GuriVR, you can find yourself writing in Spanglish. This is because (maybe intentionally, maybe because I’m too lazy) it recognizes any alias under any circumstance. In the hypothetical situation of a conflict between language keywords, I can just add a language parameter and limit the keywords, but I guarantee you that the Spanglish scenes are the funniest.

To See is to Believe

I encourage you to try to implement NLP techniques for yourself. Once you see how easy it is to improve the user experience of your apps by adding a small function with almost zero cost, you’ll become a “Low-budget NLP” fan. Writing these kinds of algorithms is also a challenge for yourself: How good can it get? How can I improve the success rate?

If you have any questions and/or want to share some of your ideas, you can find me on Twitter as @impronunciable.



Read Full Story from Source https://source.opennews.org/articles/low-budget-natural-language-processing/
This article by Dan Zajdband originally appeared on source.opennews.org on November 22, 2016 at 05:16PM

Monday, November 21, 2016

Event Roundup, Nov 21

By Erika Owens

Event Roundup, Nov 21

Talk pitches are due soon for the Internet Freedom Festival, plus a bunch of other upcoming deadlines.

Deadlines

Know of any upcoming fellowship or conference proposal deadlines? Have an upcoming event? Let us know: source@mozillafoundation.org.



Read Full Story from Source https://source.opennews.org/articles/event-roundup-nov-21/
This article by Erika Owens originally appeared on source.opennews.org on November 21, 2016 at 08:05PM

Monday, November 14, 2016

Event Roundup, Nov 14

By Erika Owens

Event Roundup, Nov 14

Apply today for ONA's leadership program for women in digital journalism.

Deadlines

Know of any upcoming fellowship or conference proposal deadlines? Have an upcoming event? Let us know: source@mozillafoundation.org.



Read Full Story from Source https://source.opennews.org/articles/event-roundup-nov-14/
This article by Erika Owens originally appeared on source.opennews.org on November 14, 2016 at 04:07PM

Friday, November 11, 2016

What You've Been Making, Nov 11

By Lindsay Muscato

What You've Been Making, Nov 11

(ProPublica)

OpenNews (and thus Source) exists to do one thing: to help a community of newsroom technologists, data scientists, and interactive designers thrive. We believe in the value of the work, now as ever. We will continue to look for new ways to support what you do, and to support you, as human beings. For now, we offer some links to your work on the way the vote went down, a map of loss, images of new Europeans, strong words, and more.

How Donald Trump Won: The Insiders Tell Their Story

(Washington Post, November 9, 2016)
An oral history and choose-your-own story of the election.

This oral history is based on four dozen on-the-record interviews with campaign advisers and other key players, conducted during the final two weeks.


Trump Nation

(USA Today November 9, 2016)
A portrait of Trump supporters from across the country.

Screen shot

Election 2016 Live Results

(Guardian, November 7, 2016)
A map that updated constantly throughout the U.S.’s election night, with results on the presidential races as well as the Senate and House.

Screen shot

Lost Cause: Seeing America Through the Losing Candidates’ Map

(ProPublica, November 7, 2016)
Some counties have supported the losing candidate in every presidential election since 1828. Here’s where they are.

Looking at these maps you can see the impact of historical trends: urbanization, party realignment, the emancipation of slaves and civil rights. You can see how red and blue states have rarely been that homogenous. You can even see how geological features created millions of years ago influence our politics today.


The New Europeans

(National Geographic, October, 2016)

The Syrian and other refugees streaming into Europe since 2015 have roiled its politics and tested its tolerance. But they’re just the latest of many waves of immigrants since World War II. Indians in Britain, Algerians in France, Somalis in Sweden—in this gallery you can meet some of the immigrants who are reshaping the continent.

Screen shot

Big Media Companies And Their Many Brands—In One Chart

(NPR, October 28, 2016)

The heated competition for our attention and for advertising dollars is increasingly pitting traditional media and entertainment stalwarts against Internet and telecom giants. And some media companies are once again seeking refuge in larger conglomerates. To wade through who owns what, we decided to build a chart that outlines some of the notable holdings of big media companies.

Screen shot

Racist,’ ‘Ignorant,’ ‘Strong’ – The Words You Used to Describe Donald Trump

(Los Angeles Times, October 31, 2016)
Words; the best words. The Los Angeles Times visualizes Donald Trump’s impact on American culture via the words used to describe the candidate.

Screen shot

Even More

A recap from Hacks Hackers Buenos Aires: rebooting journalism for real. What gerrymandering looks like, as a leafy sea dragon. What eternal hustling looks like, in baseball. Twenty-eight million scientific research articles, ready for searching. Millions of searchable documents, for journalists investigating corruption. If newspapers could vote. A chat about modern stenography, with Mirabai Knight.

Oh, and our friends at the Coral Project have launched Ask, and they’d love for you to learn more about it.

Here at OpenNews, we’re looking forward to our next community call on November 17. Let’s talk then.

Postscript

We stopped by Electionland headquarters a few hours before the first returns came in. The big open space at the CUNY School of Journalism was packed to the limits of the fire code with hundreds of journalists working elbow-to-elbow to detect and report on voting problems. Derek Willis reported in at 11pm ET on election day.



Read Full Story from Source https://source.opennews.org/articles/what-youve-been-making-nov-9/
This article by Lindsay Muscato originally appeared on source.opennews.org on November 11, 2016 at 08:10PM

Monday, November 7, 2016

Event Roundup, Nov 7

By Erika Owens

Deadlines

Know of any upcoming fellowship or conference proposal deadlines? Have an upcoming event? Let us know: source@mozillafoundation.org.



Read Full Story from Source https://source.opennews.org/articles/event-roundup-nov-7/
This article by Erika Owens originally appeared on source.opennews.org on November 07, 2016 at 05:48PM

Thursday, October 27, 2016

How (And Why) We Built A World Series Simulator

By Chris Hagan

How (And Why) We Built A World Series Simulator

(WBEZ)

This year, the Chicago Cubs evolved from lovable losers to a once-in-a-generation juggernaut.

As the Major League Baseball playoffs started in October, the team became the story everyone in Chicago was walking about. Still, we wanted to do it in a different way that could appeal both to baseball fans and those in our audience who aren’t as interested in sports.

We landed on a tool that allowed users to peek under the hood of the MLB playoffs by simulating the postseason as many times as they wanted, which we hope taught even baseball fans something new about their sport.

Why We Built It

XKCD comic

(XKCD)

Every year sites like Baseball Prospectus, Fangraphs, and FiveThirtyEight publish World Series odds for each team entering the playoffs. Those sites run thousands of simulations to publish a single percentage chance. But those numbers often don’t match up with what fans might expect.

That’s because more than other sports, baseball is ruled by randomness. In 2014 Baseball Prospectus did an article looking at what it would take for a team to have even odds to win the World Series at the start of the playoffs. Their conclusion? It would take the best team in the history of baseball.

Each series is at most seven games, which isn’t enough time for subtle advantages to show up, and with four playoff rounds those unlikely occurrences can add up quickly. Because of that, most sites gave the Cubs only a 1-in–4 shot to win the World Series, even as heavy favorites.

Our goal was to explain that reality, but do more than just report that final percentage. We wanted to let users interact with the model and experience —even for a second—what it feels like to enter a team as good as the Cubs in a tournament as random as the MLB postseason. We felt that clicking a button four times and only seeing the Cubs win once (or not at all) drove home that message better than just saying they had a “25 percent chance.”

How We Built It

The first step was to come up with a way to simulate a single game. Bill James, one of the founders of sabermetrics, came up with a formula to estimate the winning percentage of one team against another. Known as log5, the formula only needs to know each team’s regular season winning percentage to calculate one team’s expected chances against the other.

probability = (team1pct - team1pct*team2pct)/(team1pct+team2pct-2*team1pct*team2pct)

With that in hand, we started building a quick prototype in JavaScript. We wrote a function to calculate Team A’s chance of winning and compare it to a random number. If it’s greater than Team 1’s expected winning percentage, they win.

Building from there, we built a quick-and-dirty script to simulate an entire postseason. It read from a JSON file with team information, then it created objects for each of the 10 playoff teams and nine series. Each series object contained the two teams involved, the number of games to play, the number of wins a team needed to advance, and what the next series was. A "play_series" function simulated all the games until one team was victorious, and advanced to the next round (or declared them our virtual World Series champions).

Eventually I stumbled on a great article by Steve Staude in the Hardball Times that built off of James’ log5 formula using the odds ratio. Implementing Staude’s method allowed me to add home field advantage to the simulation (home teams normally win around 54 percent of the time). I also switched from using an unadjusted season winning percentage to BaseRuns, which estimates winning percentage based on the actual number of hits a team got and allowed, which smooths out the luck inherent in even in MLB’s 162-game season. But that was as complicated as our model got—nowhere near what Fangraphs or FiveThirtyEight is doing.

While simulating an individual game was straightforward, recreating the MLB rules to advance teams to the correct round and figure out home field advantage was more complicated than I first realized, even as someone who has watched as much baseball as I have. (Lesson learned: You never truly understand a thing until you have to program it in JavaScript.)

While this setup worked great—basically, it output results to the command line—the simulator needed an interface as well as a way to send the data from the simulator to the user.

I first tried to make the entire bracket with just HTML and CSS by using a Twitter Bootstrap grid, but selecting and updating each cell in the bracket using simple JavaScript objects proved too complicated. I needed to take the basic structure of my simulator and make it something that could easily modify HTML.

At WBEZ, we are big fans of the NPR Visuals team’s work. We use their Daily Graphics Rig for smaller graphics and base many of our large projects on their app template. Both make it easy to develop and publish quickly; they also make a lot of decisions so we don’t have to.

The graphics rig is based mostly on D3, and while I wasn’t originally planning on using it for this project, the NPR tools made it easier to stick with that library than to try to force the simulator into something else. Using those tools, and cribbing from an open source NCAA tournament bracket generator, I was able to rewrite my original code to fit D3. In the end, this turned out to be a really good decision.

The rewritten code uses the same JSON file of teams, and processes it to create a list of all possible series. After creating a container SVG, D3 creates an element for each series and adds a text element for the top and bottom teams in the matchup.

When a user clicks the button to simulate a series, a function clears any previous information from the bracket. Using D3, we select each of the series objects, play a series with the teams, and advance to the winner to the next series, which in turn gets simulated.

The NPR templates are also built using their pym.js, which made it easy to make the final graphic responsive. The only issue is that the graphics rig is set up to redraw the entire graphic on resize, and with dynamic data we couldn’t just reset to the original state. We solved that by splitting up the functions to simulate the series and to write the results to the bracket. When a user resizes their window, the render function redraws the bracket and then rewrites the results, but it doesn’t touch the underlying data.

In testing, we found that users might want a sense of which team was the favorite in each matchup. Calculating the series probability is more complicated that just the odds for a single game. Again, Staude’s great Hardball Times article explained how to use a cumulative binomial distribution to estimate a team’s chance in a series of any given length.

As much as I would have loved to write that in JavaScript, that went far beyond my math and programming skills. Luckily, R comes with the pbinom function, which calculates the chance of X number of occurrences of an event with Y likelihood in Z number of trials. That sounds just like a baseball series.

I built a function in R that generated the odds ratio chance of Team A winning versus Team B, then ran that through the pbinom function to make a probability matrix for all possible matchups. Finally, I imported that into the main simulator, and displayed the probability for each matchup next to the result.

As a final touch, we tried to design the graphic to match the iconic Wrigley Field scoreboard. It has the dual advantages of being easily recognizable and easy to do, with only two colors and straight lines. Thanks, 1930s designers!

The Final Score

This project was different from anything we’d tried before at WBEZ and came with a lot of moving parts. (Here's the repo.)

We had to decide if the simulator should live with our story about probability in baseball or on its own. We opted to combine them together, but I wouldn’t rule out doing it differently next time. There were also a number of extra features (such as letting users set team strength, simulating the postseason from its current state, displaying odds predictions by team, like FiveThirtyEight, et al) that we had to do without.

We saved a lot of time by using open source tools from NPR and reusing past templates. Much of the graphics code came from the graphics rig, and the bones of the design came from another story I did based around a large embed graphic last year. That helped make sure I didn’t spend a lot of development time on what was just a fun way to look at sports.

Judging by analytics and comments, our intended audience liked it, which means it was a success, given that sports don’t make up a big part of WBEZ’s daily coverage. And it gave us something we could share throughout the baseball playoffs, whether the Cubs won or lost.



Read Full Story from Source https://source.opennews.org/articles/how-and-why-we-built-world-series-simulator/
This article by Chris Hagan originally appeared on source.opennews.org on October 27, 2016 at 05:06PM

Monday, October 24, 2016

Event Roundup, Oct 24

Event Roundup, Oct 24

The Mozilla Festival is back this weekend and you can check out the schedule now.

Deadlines

Know of any upcoming fellowship or conference proposal deadlines? Have an upcoming event? Let us know: source@mozillafoundation.org.



Read Full Story from Source https://source.opennews.org/articles/event-roundup-oct-24/
This article by originally appeared on source.opennews.org on October 24, 2016 at 05:29PM

Tuesday, October 18, 2016

How NPR Transcribes and Fact-Checks the Debates, Live

By Tyler Fisher

How NPR Transcribes and Fact-Checks the Debates, Live

From the NPR transcript-fact-check combo package for the first Presidential debate

For the Presidential debate season, NPR is providing live transcripts of the debate with embedded fact checks and annotations throughout each debate night. Coordinating the workflow between live transcriptions, live fact-checking, and a live-updating page inside of our CMS was no small undertaking, resulting in what may have been our team’s most complicated technical architecture yet. I’m here to tell you all about it!

Team Structure and Workflow

On the Visuals Team, this project has combined our talents in a way we don’t always do. The core team consisted of Katie Park, Wes Lindamood, Juan Elosua, Clinton King (our current news apps intern), David Eads, and me. On the design side, Wes was responsible for design strategy and planning, and Katie was the lead designer responsible for execution of the design and user experience. On the development side, Clinton, Juan, and I all worked pretty interchangably, but Juan took the lead on parsing our live transcript and annotations and I took the lead on developing the front-end of the application. Clinton developed features on both sides of the equation. David led the team and served as product owner.

Specifically, having Wes and Katie collaborate in this way on design meant each of them could focus more on specific parts of the design process. If you divide design into “definition” and “execution,” Wes focused on definition and Katie focused on execution. That meant both Wes and Katie could explore more possibilities and, in the end, make the user experience better. On most projects, our designers have to handle both ends of the spectrum, meaning less time for exploration.

Having two designers also means there is a feedback mechanism for work that avoids “design by committee,” allowing work to move faster. Speed was important for this project, since we started working on it in earnest two weeks before the first debate.

Of course, we also collaborated heavily with our politics team in creating this system. Editor Amita Kelly was particularly indispensable in making this happen.

A Brief Technical Overview

Our application consists of six main parts, which my editor David Eads helpfully condensed into a tweet:

In a few more words, we’ve partnered with Verb8tm, NPR’s standard transcription partner, to provide us with a live transcript that they publish to an API throughout the night.

We wrote a Google Apps Script that consumes the transcript from the API, parses it into readable paragraphs, and appends it to a Google Doc.

Inside the same Google Doc, as new transcript comes in, an editor backreads the transcript and makes corrections as necessary. Fact checkers (we had over a dozen for the first debate) find statements to check and add their annotations in Google Docs Suggesting mode. A copyeditor edits both the transcript and the annotations. Overseeing all of this, our political editor Amita Kelly has the final say on what annotations go by approving the suggestions.

We have an Amazon EC2 server running a daemon that downloads the full Google Doc as an HTML file, parses it with copydoc, our library for cleaning the HTML from a Google Doc, then executes some custom debate-specific parsing code, and finally publishes a clean HTML file to Amazon S3.

Then, our client-side app (also hosted on Amazon S3) consumes that HTML, turns it into a virtual DOM element, and diffs it with the current state of the DOM to apply only the necessary changes to the document.

Finally, we embed this client-side app inside of our CMS, which we handle with our responsive iframe library, pym.js, and a whole lot of custom cross-domain iframe messaging to handle the animation when a user scrolls to see a new animation. We are embedding this app not only inside of npr.org, but also on member station websites across the country. To achieve this, we use the new pym-loader JavaScript Juan Elosua developed for pym version 1.0.0.

We haven’t open sourced our code yet, unfortunately. There’s some work that needs to be done to remove some private keys and things, and we, like many of you, are too busy with the rest of our elections work for the next few weeks. We promise to get it open sourced eventually, but note that having the full source probably wouldn’t help you. Our setup is custom-built around our relationship with our transcription partner, who transcribes all of NPR’s content every day. So you’re more likely to get more out of reading this post than straight-up using our code.

The Nitty Gritty

I know, I know. You’re hungry for the details. Let’s follow the same path I outlined above, from the transcript to the embedded widget.

Transcript API

Our partners at Verb8tm (whose technical support staff were among the most cooperative partners we’ve ever worked with) provided an API endpoint that returns the latest version of the transcript when requested. The endpoint publishs the transcript as an SRT file, a plain-text format for subtitles. A subtitle block comes in four parts:

  1. A number representing the place of the subtitle in the sequence of all subtitles
  2. The begin and end time of the block of text relative to the start of the subtitle
  3. The subtitle
  4. A blank line breaking subtitles apart

The API also returns a header that indicates the final caption identifier (piece one of the SRT block) in the current version of the SRT. That means we can store that number and only request anything that came after that subtitle block on subsequent requests.

The Google Apps Script

Google Apps Scripts are funny. They are essentially self-contained JavaScript environments that provide extended functionality to interact with Google services, such as, you guessed it, Google Docs. Google Apps Script projects connect with your Google account’s Google Drive. Within a project, you can have multiple scripts that all exist within the same environment. For example, we have a file that deals with parsing the SRT file, and we have another that deals with appending the parsed transcript to the Google Doc.

Using a Google Apps Script is the only way to append our new transcript chunks to a Google Doc, but it also comes with certain limitations.

In a Google Apps Script, you can create time-based triggers, meaning code can execute on a timer. It works essentially like a cron job, and just like a cron job, you can only execute once a minute. That means we can only get new transcript from the API once a minute.

So, every minute, our Google Apps Script hits the Verb8tm API. The first time it runs, it gets the entire available SRT available. It stores the last SRT ID for the next time the function runs. On subsequent runs, it sends that ID back to the API, and the API returns only the new parts of the SRT.

Once we had our SRT chunks, the Google Apps Script also has to parse SRT into readable paragraphs. Luckily, Anton Bazhenov wrote a JavaScript parser that turns an SRT block into an object a long time ago, so we use that in the script.

Next, with our parsed SRT, we have to build intelligible paragraphs. There are pieces of conventional SRT syntax that we rely upon. Particularly, if a subtitle introduces a new speaker, the block is preceded by >>, so when Lester Holt began speaking at the beginning of the first debate, the SRT block looked something like:

1
00:00:000 --> 00:06:454
>>LESTER HOLT: I'm Lester Holt anchor of 
NBC nightly news...


We use these new speaker indicators to break up the SRT chunks into paragraphs. Since the SRT chunks are not broken into paragraphs or sentences or any predictable grammatical structure, the only thing we can rely upon is the introduction of a new speaker. So if a new SRT chunk does not begin with a speaker, we append the new chunk to the end of the preceding paragraph.

Once we’ve built our paragraphs, we append them to the Google Doc using Google Apps Scripts’ Document Service. This allows us to append to the document easily.

Remember, however, that we have fact checkers working live in the document. So let’s say, for example, that the last SRT chunk we received ended halfway through Donald Trump’s answer to a question on jobs. The next time we receive an SRT chunk, we want to append the rest of Trump’s answer to the same paragraph. But what if someone has already started fact checking that paragraph?

You’re starting to see some of the limitations of this rig. We can’t have fact checkers work on the last paragraph of the transcript. We handle this by automatically adding a horizontal rule and a line of text that says “DO NOT WRITE BELOW THIS LINE,” a small piece of UI that reminds fact checkers. It’s not a huge deal, since new transcript comes in every minute, but it is important to note that this isn’t a perfect system. But letting our fact checkers work inside a Google Doc has been worth the downsides.

We also built a good amount of tooling around the development of Google Apps Scripts. Google would very much like you to work inside of their editor on the web, but through the Google Drive API, we were able to get our scripts inside of our own dev environment and version control, and upload them to Google Drive for execution.

Google Apps Scripts also provides an execution API, which allows you to fire a function inside of a Google Apps Script from a third party, such as a Fabric command. This allows us to set configuration on the Google Apps Script project, such as telling the script what the ID of the Google Doc we want to append to is.

Unfortunately, the execution API cannot fire functions that create triggers (the script’s version of a cron job), so actually starting the trigger is a manual process that requires going into the Google Apps Script on Google Drive.

Google Apps Scripts comes with built-in logging, but it is not very good. Instead, we used our own logging system heavily based on BetterLog by Peter Herrman, which sends messages to a Google Spreadsheet.

The Google Doc

Again, as the transcript is appended to the document, we have fact checkers adding annotations within the same document, as well as editors cleaning up and copyediting the transcript itself. This creates a potentially chaotic environment in the document, and one that could lead to serious errors or incomplete statements from our reporters appearing on the page.

The secret ingredient to the Google Doc workflow is Suggesting mode. By default, anyone we share the document with can only work in Suggesting mode, with the exception of the editors who need the ability to edit and approve suggestions. Working in Suggesting mode means that fact checkers can construct their entire annotation, and our editors can fully edit them, before any of it reaches the exported HTML of the document.

Using Conventions to Create Structure

Because Google Docs is essentially a word processor, it’s a challenge to create the basic structure required to show which candidate statements we are fact checking, as well as to distinguish annotations from the transcript text itself.

Let’s look at an example from our first debate document:

DONALD TRUMP: Excuse me. I will bring back jobs. You can’t bring back jobs.

HILLARY CLINTON: Well, actually, I have thought about this quite a bit.

DONALD TRUMP: Yeah for thirty years.

HILLARY CLINTON: Well, not quite that long. I think my husband did a pretty good job in the 1990s. I think a lot about what worked and how we can make it work again.

DONALD TRUMP: Well, he approved NAFTA. He approved NAFTA, which is the single worst trade deal ever approved in this country.

HILLARY CLINTON:…a balanced budget, and incomes went up for everybody. Manufacturing jobs went up also in the 1990s, if we’re actually going to look at the facts.

NPR: sh-jobs-year For comparison: jobs per year was strongest under Bill Clinton (2.8 million), followed by Carter (2.6 million), Reagan (2 million), Obama (1.3 million as of January), H.W. Bush (659,000), and W. Bush (160,000).

And here’s the same section as presented in our app:

display in the app

We add two things to the document to handle annotations. First, we bold the text we want highlighted in the transcript. This is simple enough. The next part—unique IDs for each annotation—is harder.

Our annotations are always prefixed with NPR: in the same way that HILLARY CLINTON: and DONALD TRUMP: appear in the transcript. Following the prefix, we give every annotation a slug. The first part of the slug is the author’s initials. Then, any number of words can be appended to the slug, we just need the slug to be unique. In the client-side app, we rely upon unique IDs for each annotation for a number of things including triggering animations and tracking read fact checks in our analytics. This will allow us to build unique shareable URLs for each annotation in the future.

Unfortunately, because an annotation can be added to any part of the document at any time, we cannot build IDs sequentially. We also cannot build IDs based on the text of the annotation because the annotations may get edited further after they first appear on the site. The only way to guarantee we maintain the same ID for any annotation is to hard-code the annotation within the Google Doc. The obvious risk here is that duplicate IDs are created, so we check for that when we parse on the server later.

The Server

Our server does three things: it uses the Google Drive API to download our transcript Google Doc as HTML, runs that HTML through copydoc and a custom parser to construct the HTML we use on the client side, and deploys that HTML to Amazon S3. This is all set up as a Flask app controlled by a daemon running on Ubuntu through upstart.

The custom parser handles a few specific cases:

  1. Takes speaker attributions (i.e. DONALD TRUMP:), removes them from the paragraph and recreates them as <h4> tags above the paragraph.
  2. Replaces <strong> tags with custom <span> tags that handle our highlighted text.
  3. Finds annotations and creates our custom DOM structure for annotations. This includes matching the author’s initials in the slug to a dictionary that contains their full name, job title, and a link to their photo.

This is all handled by parsing the HTML through Beautiful Soup. Transforming the annotations is the most complicated case. Here, we have to identify annotations based on the NPR: slug syntax explained in the Google Doc section. Then, in the function below we recreate the contents of the annotation and wrap it inside of the markup structure we want our annotations to have on the client.

extract_fact_metadata_regex = re.compile(ur'^\s*(<.*?>)?NPR\s*:\s*(([A-Za-z0-9]{2,3})-[A-Za-z0-9-]+):?\W(.*)', re.UNICODE)


FACT_CHECK_TPL = '''
<div class="annotation" id="%(slug)s">
    %(fact_check_text)s
    <div class="annotation-header">
        <img class="%(annotation_label_class)s" src="%(author_img)s"></img>
        <div class="annotation-byline">
            <a class="byline-name" href="%(author_page)s">%(author_name)s</a>
            <span class="byline-role">%(author_role)s</span>
        </div>
    </div>
</div>
'''
PARAGRAPH_TPL = '<p>%s</p>'

def transform_fact_check(paragraphs, doc):
    """
    adds markup to each fact check
    """

    # This will wrap all the fact check paragraphs
    fact_check_wrapper = doc.soup.new_tag('div')
    fact_check_wrapper['class'] = 'annotations-wrapper'

    for paragraph in paragraphs:
        # We need to recreate the contents since copyDoc
        # is stripping the spans but leaving the child structure untouched
        combined_contents = ''
        for content in paragraph.contents:
            combined_contents += unicode(content)
    m = extract_fact_metadata_regex.match(combined_contents)
    if m:
        slug = m.group(2).lower()
        annotation_label_class = 'annotation-label'
        author = m.group(3).lower()
        if m.group(1):
            clean_text = m.group(1) + m.group(4)
        else:
            clean_text = m.group(4)

        # Grab info from dictionary
        try:
            author_name = app_config.FACT_CHECKERS[author]['name']
            author_role = app_config.FACT_CHECKERS[author]['role']
            author_page = app_config.FACT_CHECKERS[author]['page']
            author_img = app_config.FACT_CHECKERS[author]['img']
        except KeyError:
            logger.warning('did not find author in app_config %s' % author)
            author_name = 'NPR Staff'
            author_role = 'NPR'
            author_page = 'http://www.npr.org/'
            author_img = ''

        # Handle author images, which some authors may not have
        if author_img == '':
            annotation_label_class += ' no-img'
    else:
        logger.error("ERROR: Unexpected metadata format %s" % combined_contents)
        return None
    new_paragraph = BeautifulSoup(PARAGRAPH_TPL % (clean_text), "html.parser")
    fact_check_wrapper.append(new_paragraph)

    fact_check_markup = FACT_CHECK_TPL % {
        'slug': slug,
        'annotation_label_class': annotation_label_class,
        'author_page': author_page,
        'author_name': author_name,
        'author_role': author_role,
        'author_img': author_img,
        'fact_check_text': fact_check_wrapper}
    markup = BeautifulSoup(fact_check_markup, "html.parser")

    return markup

Once we have run the entire transcript through our parser, the HTML string returned through Beautiful Soup is rendered to an HTML file through a Flask route. We bake that route to a flat HTML file and deploy it to S3, where our client app will request it and display it to the user.

The Client-Side App

Our client-side app handles two main things:

  1. Requesting the flat, parsed HTML file that the server deployed to S3, parsing it into the virtual DOM, and patching the DOM with the necessary changes.
  2. Managing the state of each annotation—that is, whether a user has seen or read a particular annotation.

I’m a framework nihilist, so rather than use a heavy framework with all kinds of assumptions about code structure like React, we are using virtual-dom with its bindings for hyperscript to construct virtual DOM elements and handle the diffing needed to apply only the necessary changes to the DOM.

Our app has three main parts: the header, the transcript, and the footer. Our initial HTML page is quite simple:

<html>
    <head>
    <!-- CSS and meta tags and whatnot -->
    </head>
    <body>
        <div class="header-wrapper"></div>
        <div class="transcript-wrapper"></div>
        <div class="footer-wrapper"></div>
    </body>
</html>

On page load, we render our initial DOM objects for the header, transcript, and footer. We do that by first constructing each piece as a virtual DOM element in hyperscript, rendering it to a real DOM element, and then appending the real DOM element to the page.

const initUI = function() {
    const headerData = {
        'updated': moment().format('hh:mm a') + ' EDT ',
        'numAnnotations': 0
    }
    headervDOM = renderHeadervDOM(headerData);
    headerDOM = createElement(headervDOM);
    headerWrapper.appendChild(headerDOM);

    transcriptvDOM = renderInitialTranscriptvDOM();
    transcriptDOM = createElement(transcriptvDOM);
    transcriptWrapper.appendChild(transcriptDOM);

    const footerData = {
        'updated': moment().format('hh:mm a') + ' EDT ',
        'newAnnotations': 0
    }
    footervDOM = renderFootervDOM(footerData);
    footerDOM = createElement(footervDOM);
    footerWrapper.appendChild(footerDOM);
}

To see virtual DOM construction in action, here is how we render the footer:

const renderFootervDOM = function(data) {
    const annoStr = data.newAnnotations === 1 ? 'annotation' : 'annotations';
    const annoNum = data.newAnnotations === 0 ? 'no' : data.newAnnotations;

    return h('div.footer', [
        h('div.update-wrapper', [
            h('p.update-notice', [
                'While you were reading, we added ',
                h('span.update-number', annoNum + ' new ' + annoStr + '.'),
            ]),
            h('p.update-info', 'To make new annotations easier to spot, we\'ve marked them in yellow. Scroll back up to see what you\'ve missed.')
        ]),
        h('div.footer-nav', [
            h('span.last-updated', ['Last updated: ' + data.updated]),
            h('a.jump-to-top', {
                href: '#'
            }, 'Back to top')
        ])
    ])
};

With the UI initialized, we can request our transcript file. We’re using superagent and setting an “If-Modified-Since” header on the request to get new data only when it is available. When we request the transcript, we receive the HTML as a string in the response. And here’s where the magic happens.

New to me was JavaScript’s native DOM parser, which can take an HTML string and turn it into a DOM node. We use that to turn our response into a DOM node and parse it to our needs.

It’s not as simple as just diffing the changes to the DOM with the current DOM on the page. This is because we animate annotations that a user has not read yet, which means we have to maintain state on the client. We handle those animations through CSS classes. So, when the entire transcript is parsed, we need to determine which annotations need to have a class of unread and which annotations do not.

Consider this function:

const buildTranscriptvDOM = function(transcript) {
    const children = transcript.children;
    const childrenArray = Array.prototype.slice.call(children);
    return h('div', {
        className: transcript.className
    }, [
        childrenArray.map(child => renderChild(child))
    ]);

    function renderChild(child) {
        let element = null;
        if (child.tagName === 'DIV') {
            if (child.classList.contains('annotation')){
                element = renderAnnotation(child);
            }
        } else {
            element = virtualize(child);
        }
        return element;
    }

    function renderAnnotation(child) {
        const id = child.getAttribute('id');
        if (readFactChecks.indexOf(id) === -1){
            child.classList.add('unread');
        }

        if (seenFactChecks.indexOf(id) === -1){
            child.classList.add('offscreen');
        }
        return h('div', {
            id: child.getAttribute('id'),
            className: child.className
        },[
            virtualize(child.querySelector('.annotations-wrapper')),
            virtualize(child.querySelector('.annotation-header'))
        ])
    }   
}

A few things to note: the virtualize() function comes from vdom-virtualize, which transforms a DOM node into a virtual DOM node. This is necessary so we can apply virtual-dom’s diffing engine. And we are maintaining global lists of fact checks a user has read (had on screen for about a second and a half) as well as fact checks a user has seen at all.

This function takes the transcript DOM node, finds annotations, and decides whether or not they have been read or seen by the user yet, and applies the necessary classes. It also transforms the entire DOM node into a virtual DOM node.

So, yes, we are taking an HTML string from an external file, turning it into a DOM element, then virtualizing it (after some reconstruction), then unvirtualizing it and applying to the DOM. It’s roundabout. But it works. (If I had to do it again, I might look at morphdom instead.)

Let’s step back. We have requested an HTML file that contains the parsed transcript from the Google Doc. Once we have a response from that file, we update the transcript on the page. That high-level function looks like this, including a call to our function above:

const updateTranscript = function(data) {
    const domNode = parser.parseFromString(data, 'text/html');
    const transcript = domNode.querySelector('.transcript');
    const newTranscriptvDOM = buildTranscriptvDOM(transcript);
    const patches = diff(transcriptvDOM, newTranscriptvDOM);

    transcriptDOM = patch(transcriptDOM, patches);
    transcriptvDOM = newTranscriptvDOM;
    registerTrackers();
}

There’s a couple pieces of virtual-dom magic here, notably the diff() and patch() functions. diff() takes two virtual DOM nodes and finds the differences between them, much like a git diff, and returns patches. patch() takes those patches and applies them to a real DOM node.

Based on our updated transcript DOM, we update our header and footer based on, basically, the lengths of document.querySelectorAll() calls. We use the same virtual DOM diffing and patching logic as above.

Embedding

The final piece of this monstrosity is making the whole thing work inside an <iframe>. We have pym.js, our JavaScript library for embedding iframes responsively, which gets us most of the way there. But much of the functionality of the transcript—controlling when an annotation animates and when it fades to gray—hinges on its visibility in the viewport. Iframes, especially cross-domain iframes in our case, make this more complicated.

Detecting the visibility of an element inside an iframe from the parent when the iframes are not on the same domain is not possible. Similarly, detecting the scroll position of a user from inside the child iframe is not possible. However, Pym gives us the ability to send messages cross-domain, so we can leverage that functionality to complete a handshake of sorts. Here’s how it works.

When we identify a new annotation in our virtual DOM-diffing, we send a message to the parent asking it to register a visibility tracker for that annotation. Our tracker listens for scroll events and checks to see if its annotation is in the viewport. It does this by sending a message through Pym to the child. When the child receives the message, it finds the size of the annotation and its position relative to the iframe with getBoundingClientRect(). Then, it returns that rect to the parent. With that information, the parent calculates the user’s scroll position and the annotation’s true position on the page by offsetting the child’s results with the iframe’s current position on the page. With all this information, it can determine if the annotation is visible. Phew!

Here’s some simplified code to give an example.

On the parent:

var VisibilityTracker = function(iframe, id) {
    var WAIT_TO_ENSURE_SCROLLING_IS_DONE = 40;

    function isElementInViewport(rect) {
        var iframeRect = iframe.getBoundingClientRect();
        var vWidth   = window.innerWidth || document.documentElement.clientWidth;
        var vHeight  = window.innerHeight || document.documentElement.clientHeight;

        var verticalScroll = vHeight - iframeRect.top;
        var bottomBound = rect.bottom + vHeight;

        // Track partial visibility.
        var leftSideIsToRightOfWindow = rect.left > vWidth;
        var rightSideIsToLeftOfWindow = rect.right < 0;
        var topIsBelowVisibleWindow = rect.top > verticalScroll;
        var bottomIsAboveVisibleWindow = verticalScroll > bottomBound || rect.bottom > verticalScroll;

        if (leftSideIsToRightOfWindow  ||
            rightSideIsToLeftOfWindow ||
            topIsBelowVisibleWindow   ||
            bottomIsAboveVisibleWindow) {
            return false;
        }

        return true;
    }    

    function checkIfVisible(rect) {
        var newVisibility = isElementInViewport(rect);

        if (!isVisible && newVisibility) {
            pymParent.sendMessage('fact-check-visible', id);
        }

        isVisible = newVisibility;
        return newVisibility;
    }


    addEventListener('scroll', handler, false);
    pymParent.onMessage(id + '-rect-return', function(rect) {
        checkIfVisible(rect);
    });

    function sendRectRequest() {
        pymParent.sendMessage('request-client-rect', id);
    }

    var handler = throttle(sendRectRequest, WAIT_TO_ENSURE_SCROLLING_IS_DONE);

    // Initialize
    sendRectRequest();
}

// register the tracker
pymParent.onMessage('new-fact-check', function(id) {
    var iframe = document.getElementById('transcript-embed');
    var tracker = new VisibilityTracker(iframe, id);
    trackers[id] = tracker;
});

And on the child:

let trackedFactChecks = [ ];

pymChild.onMessage('fact-check-visible', onFactCheckVisible);
pymChild.onMessage('request-client-rect', onRectRequest);

const onFactCheckVisible = function(id) {
    const factCheck = document.getElementById(id);
    factCheck.classList.remove('offscreen');
    if (seenFactChecks.indexOf(id) == -1) {
        seenFactChecks.push(id);
        updateFooter();
    }
}

const onRectRequest = function(id) {
    const factCheck = document.getElementById(id);
    const rect = factCheck.getBoundingClientRect();
    pymChild.sendMessage(id + '-rect-return', rect);
}

// this function fires when we receive new transcript
const registerTrackers = function() {
    const factChecks = document.querySelectorAll('.annotation');

    let currentFactChecks = [ ];

    [ ].forEach.call(factChecks, function(check) {
        const id = check.getAttribute('id');
        currentFactChecks.push(id);

        if (trackedFactChecks.indexOf(id) === -1) {
            trackedFactChecks.push(id);
            pymChild.sendMessage('new-fact-check', id);

            check.classList.add('new');
            setTimeout(function() {
                check.classList.remove('new')
            }, 200);

            // this uses pym to update the iframe height
            check.addEventListener('transitionend', updateIFrame);
        }
    });
}

This project also gave us an opportunity to add a new feature to Pym. We wanted the ability to scroll the page to certain elements within the iframe, so we added two shortcuts that make this possible: scrollParentToChildEl and scrollParentToChildPos. This lets you tell the parent to scroll to a child element (referenced by its ID) or a pixel position relative to the top of the child.

The Recap

I’ve covered the technical side of our transcripts and fact checking, but I haven’t even approached the design decisions we made and why we made them—topics my colleagues Katie Park and Wes Lindamood can dig into.

Just to go over all the things here, I’ll bring back David’s tweet.

transcription service ←→ google app script → google doc (+18 factcheckers) ←→ server → s3 → embedded widget

This is a highly complicated piece of technology, but ultimately, we designed it so we could get a dozen-plus fact checkers doing what they do best, in an editing environment that makes sense to them. And the results showed. The first debate led to NPR’s biggest traffic day ever, and this is the most successful piece—in terms of traffic—the Visuals team has ever built.



Read Full Story from Source https://source.opennews.org/articles/how-npr-transcribes-and-fact-checks-debates-live/
This article by Tyler Fisher originally appeared on source.opennews.org on October 18, 2016 at 03:51PM

Monday, October 17, 2016

Event Roundup, Oct 17

By Erika Owens

Deadlines

Know of any upcoming fellowship or conference proposal deadlines? Have an upcoming event? Let us know: source@mozillafoundation.org.



Read Full Story from Source https://source.opennews.org/articles/event-roundup-oct-17/
This article by Erika Owens originally appeared on source.opennews.org on October 17, 2016 at 04:55PM

Friday, October 14, 2016

Source Project Roundup, Oct 14

By Lindsay Muscato

Source Project Roundup, Oct 14

(Washington Post)

Here’s a look at what we’ve been reading and scrolling through lately: border issues, complaints against cops, campaign data, invisible labor, and more.

Fenced Out

(Washington Post, October 14, 2016)
A beautifully wrought, immersive journey into the current refugee crisis in Europe, chronicling the region’s fences, fears, and frustrations.


Over 125K complaints against more than 25K Chicago cops

(Chicago Tribune, October 14, 2016)

The Tribune’s analysis of thousands of pages of internal police records — handed over by Mayor Rahm Emanuel’s administration Wednesday — offers an unprecedented look at complaints against Chicago police dating to 1967, making it possible for the first time to identify officers with a long history of complaints.


Kids in Prison: Getting Tried as An Adult Depends on Skin Color

(WNYC, October 10, 2016)

Kids who commit certain crimes in New Jersey, like robbery, drug trafficking or homicides, can be tried as adults. Their mugshots and criminal records are made public and they face the same, long prison sentences as adults. But getting “waived up” into the adult system won’t happen unless a prosecutor requests it. And according to an analysis by the WNYC Data News Team, most of their requests are for black kids…"

Screen shot

More Than 160 Republican Leaders Don’t Support Donald Trump. Here’s When They Reached Their Breaking Point

(New York Times, October 9, 2016)
Here’s just one of the incredible projects detailing the latest developments in the U.S. presidential election. This one shows the long, slow slide away from Donald Trump, by members of the G.O.P.


FEC Itemizer

(ProPublica, updated regularly)

Itemizer allows you to browse electronic campaign finance filings from the Federal Election Commission and to see individual contributions and expenditures reported by committees raising money for federal elections.

Screen shot

La Fuerza Laboral Invisible de N.J.

(NJ.com, September 18, 2016)
This investigation, conducted with assistance from Reveal, shows widespread racial and gender discrimination in temp agencies that supply blue-collar workers. Published in Spanish as well as English.

Even More

The pitfalls of relying on zip code borders, when examining public health data in places like Flint. You might not need JS. Testing, testing, testing for d3-force. Riddles and more riddles. A math bot for scientists. A chat bot for refugees.

Here at OpenNews, we had a great Community Call earlier this month, where we learned about the future of Sunlight Labs and dove into a great interactive from the Texas Tribune. If you missed it, fear not, we saved you the notes. We’re also looking for volunteers to help us build a playbook for open-sourcing newsroom projects. Apply by Monday to join us for two days in December.

Our next Community Call is October 20, and we’d love to see you there.



Read Full Story from Source https://source.opennews.org/articles/source-project-roundup-oct-14/
This article by Lindsay Muscato originally appeared on source.opennews.org on October 15, 2016 at 12:03AM

Wednesday, October 12, 2016

Practical Tips for Improving Mental Health in the Newsroom

By Joel Eastwood

Practical Tips for Improving Mental Health in the Newsroom

Everyone needs it. (Internet Archive)

Every year, roughly one in five American adults experiences a mental illness. These range from anxiety and mood disorders to depression, bipolar disorder and schizophrenia. In extreme cases, mental illness can lead to suicide, the 10th leading cause of death in the U.S. and the second-highest in the 10 to 34-year-old range.

Working in a newsroom poses particular challenges to mental health: the job typically involves a high-stress environment, long hours, tight deadlines, exposure to graphic images and videos, and an unstable industry with uncertain benefits and job security.

Mental illness is exacerbated when it is not discussed openly or taken seriously. This July in Portland at SRCCON, dozens of journalists, developers, and newsroom workers sat down together to share their personal experiences with mental health.

Those discussions have been summarized into this list of practical tips intended for anyone working in a newsroom concerned about their own mental health or the well-being of their colleagues and employees.

Some of these suggestions are achievable by a single person. Others require a change in behavior from a group of colleagues, everyone at a publication, or even the entire industry. Some of these ideas require additional time or money, which may make them unfeasible depending on your personal circumstances. But we hope this list starts a crucial conversation.

Caring for Yourself

Mental health starts with you.

  • Recognize what mental illness looks like. Deteriorating mental health can manifest in many different ways. Learn to spot the symptoms in yourself and your coworkers with resources like this, from the National Institute of Mental Health.

  • Practice self-care: be diligent and unapologetic about taking time for yourself. At work, this means you should try to take regular breaks where you step away from your desk or assignment.

  • Outside of work, set aside time for activities and hobbies that take your attention off work.

  • Frame self-care as a productive investment in your well-being, not a chore or drain on your time. Unhealthy habits such as sleep deprivation or a poor diet are short-term shortcuts that can have long-term consequences.

  • Daily exercise, such as a short walk, can help reduce stress.

  • Learn to say no. Know your limits and learn when you can turn down additional work or step away from additional responsibilities.

  • Don’t read the comments. The internet has made it easier than ever for anyone to leave negative, offensive, and hurtful commentary. Don’t engage with it if you don’t have to.

  • On the other hand, keep a record of positive social media messages or reader emails. Refer to this file as a reminder that real people are reading or watching your stories, and your work makes a positive difference in their lives.

  • Don’t struggle in silence. Have at least one person—a coworker, a friend, a partner or a family member—with whom you can be honest about your mental state and any difficulties you may be having.

  • Have a newsroom ally you can talk to about specific problems at work, or just someone to vent to.

  • If possible, reduce or eliminate the amount of work you take home. Try to limit checking email to work hours. One reporter encrypted his email for security and was pleasantly surprised to learn it meant he could no longer check email on his phone, forcing him to leave work at the office.

  • If bringing your work home is inevitable, set aside a space in your house—e.g. your bed, your living room, your garden—where you do not work. Enforce those boundaries.

  • Limit the number of distractions that can interrupt you. For example, reduce the number of apps on your phone that have the ability to send notifications. Instead, consciously choose when to devote attention to them.

  • If you’re comfortable, talk openly about how graphic content is affecting you. Acknowledge your emotions, speak about your suffering, and communicate with your coworkers.

  • Accept imperfection. There’s only so much work any person can do on a tight deadline with limited resources.

  • Be okay with cuts and edits to stories. Remember that readers will never see your perfect, imaginary version of the story, or know about the quote that didn’t make the cut or the interview you didn’t get.

  • Accept your own limitations. It’s easy to be idealistic and try to change the world or right injustices with every story; sadly, that is an impossible standard. Accept your own limitations and focus on telling one story at a time.

  • If you are in crisis, call a suicide prevention hotline: 1–800–273-TALK (8255)

As a Manager

Policies and practices to help those who work for you.

  • Don’t wait until there’s a crisis. As a manager, make sure you and your employees are well-versed in your organization’s benefits and policies. Just as you brief new hires about sick days and overtime policies, inform them what benefits the publication has available for mental health.

  • Depersonalize the issue of mental illness to avoid singling out any one person. It’s not a specific person who gets sick; people, in general, get sick.

  • Be a model of self-care. Perpetually working evenings and weekends can set an unrealistic example for your teammates and discourage them from safeguarding their own health.

  • Pay attention to the patterns of your coworkers and employees. Are they taking on too much, working evenings and weekends? Are they regularly missing mealtimes? Are they behaving differently?

  • If possible, try to accommodate requests for flexibility, such as when employees need to take time to visit the doctor, fill a prescription, or work from home.

  • Recognize that different employees will have different needs based on their personality, lifestyle, and work habits; there is no one-size-fits-all approach. Ask your employees questions such as, “How do you prefer to receive feedback?” or “Are you comfortable working overtime on weeknights?” to understand their individual needs.

  • Encourage strategies and workflows that increase communication and teamwork such as mentorship, pair programming, and story reviews.

  • Create a clear line of communication between management and your employees, which they are comfortable using to discuss sensitive topics.

Newsroom Culture

How to build better, kinder, more accommodating work environments.

  • Establish a physical space (e.g. break room) or electronic space (e.g. Slack channel) which provides a mental break from the ceaseless onslaught of news.

  • Establish a principle of “never one”: don’t allow a difficult role or responsibility to fall entirely on the shoulders of one person. Provide them support so they’re not isolated in their task, and everyone has peace of mind knowing there’s a backup plan for a critical role.

  • Rotate duties and assignments so one person is never permanently stuck in a particularly difficult role or beat, such as reviewing graphic images or covering terror attacks.

  • Acknowledge that trauma and PTSD can affect reporters and photographers who bear witness to traumatic events. They can also be felt by those who experience trauma secondhand through images, videos, and descriptions.

  • Recognize when coping mechanisms for stress, such as drinking and smoking, are becoming unhealthy. Newsroom culture can sometimes tolerate or even encourage self-destructive behavior, which is epitomized in the archetype of the hard-drinking newspaper reporter.

  • Organize different types of newsroom events, not just ones that involve drinking. Provide different avenues for employees to socialize if not everyone is comfortable drinking.

More Reading



Read Full Story from Source https://source.opennews.org/articles/practical-tips-improving-mental-health-newsroom/
This article by Joel Eastwood originally appeared on source.opennews.org on October 12, 2016 at 06:00PM

Latest Posts