Brown Bag Presentation: Javascript MV*

The Rails test environment ships with controller caching disabled, which you might not prefer. Otherwise, it may be useful to be able to toggle it on/off during the test run. To provide an optional caching mechanism for your specs, configure an around block:

The around block takes the RSpec example object as an argument. The block is triggered when :caching is detected as a key in an example’s metadata. The example object provides a number of methods for test introspection, allowing you to make changes before and after calling #run to execute the spec. Here, we are storing the previously set value of ActionContoller::Base.perform_caching, setting it for the local suite, and setting it back to the original value after it completes.

As a result, we now have a simple, explicit mechanism for introducing caching to individual specs and suites:

Let’s imagine we have an application with a User model (might not be a stretch) and we’re building a page that loads a bunch of user gravatars. Our User model has accumulated a ton of attributes, but we may only need a limited set to render the gravatar properly. We might add a line like this in the controller action to select only the needed attributes:

@users = User.select(%w[ id name gravatar_id ]).all

Perhaps to make this more readable, we’ll extract our select statement into a scope in the User model:

# user.rb
scope :select_gravatar_attributes, select(%w[ id name gravatar_id ])

# controller action
@users = User.select_gravatar_attributes.all

We’ll incur less memory overhead during this action at runtime since the app won’t have to load as much data. The caveat is that our @users won’t have access to the data we left behind, so we need to take care only to call methods that rely on the data we have actually loaded. It would be easier to test our lighter users if we could treat them as a different type.

Applying the model-slimming refactoring we discussed in part 1, we can extract gravatar related functionality into a UserExtensions::Gravatar module and create a Gravatar::User that inherits from our User model. Furthermore, we can change our select_gravatar_attributes to a default_scope in the subclass, so whenever we grab Gravatar::User from the db, it’ll have only the attributes we need. Our controller action would now look like this:

@users = Gravatar::User.all

To put our savings to the test, we can run some basic benchmark tests. I created a sample rails 3.1 app and created ~2000 users records in my development database and ran the rails benchmarker script to load all users both normally and as gravatars. Here are the results:

All users

All as gravatars

The memory footprint for gravatar users is half of that for our ‘fat’ users. Obviously results will vary greatly depending on architecture, application and the attributes selected, but this anecdotal case demonstrates potential for some big wins in instantiating slimmer subclasses of your heavy models.

But what happens when your models grow too large? It’s common to have one or two models (Users and Groups, anyone?) carry a large burden. Over time, we see Ruby files approaching and surpassing thousands of lines and their associated tables accumulating column excess. Before you know it, it’s time to put your models on a diet. It’s useful to have some patterns in your tool-belt to start trimming the fat.

A good refactoring step is to pull out modules of related behavior. In the User model, for example, we might extract modules for related instance methods, associations and validations for functional scopes like admin access or group memberships. I prefer to namespace the modules under UserExtensions so it’s clear, at least in this case, that these are meant for composition of the User model rather than for sharing code across objects. Now ActiveSupport::Concern makes it super-simple to define both instance and class methods in your module and ensure they’re included properly. (Some good reading on better ruby idioms for mixing in class and instance methods).

One convention is to place the modules in “#{model_name}_extensions” folders in your app/models directory to group the code logically. For example, UserExtensions::Admin would be located at app/models/user_extensions/admin.rb. Breaking out the code in to functional chunks like this is a good first step in downsizing your model files. It also more easily allows methods and associations to be extracted into other classes/tables.

Added: Check out part 2 of this post for thoughts on slimming down our models at runtime.

A few impressions:

Eris Reis demonstrated how Ghostbusters is about a lean startup at its core… guys start something on the side, try to turn it into a business from nothing, get their first customer just when the money is about to run out, interesting things start to happen as the business grows. Perhaps the GBs were lucky that Zul arrived when he did, but real startups don’t have that luxury… instead of “waiting for Zul”, lean startups need to be more flexible and proactive…

Farrah Bostic quit her job at an ad agency after trying to apply lean methodology to a business model that wasn’t set up to interact with customers that way…

When startups try to do much, it’s hard to do much well. It may be better to pick one idea to focus on, or as Brittany Laughlin explained about her experiences with here social travel app gtrot.com, “It’s better to be young and simple, than young and stupid.”

It was inspiring to hear that Yipit took off after a pivot that took days to turnaround. Their first attempt took 9 months to implement, the next, 3 months, but the idea that stuck was built in 3 days at first. They proved their idea with an initially low-tech solution–including an late night “crawler” that involved crawling out of bed at 3 am nightly to copy and paste data–that they were able to build into something more robust later with more confidence. Founder Vin Vacanti challenged the audience, “If we can prove a business idea in 3 days, why aren’t you doing it too?”

Giff Constable provided a retrospective on one of his failed ideas: not enough resources, picking a difficult problem… His talk reinforced the lean notion that it’s okay to fail; the knowledge gained from our failures is invaluable to our future efforts.

Shout outs to NYC Lean Startup for running a successful event and to Pivotal Labs for hosting.

A fundamental shift in Rails 3 is html escaping by default. A number of posts describe the need for this change, an important tool against cross-site scripting (XSS) attacks. Rather than rely on developers to escape user content where needed, all strings that don’t originate from Rails (link_to, form_for, etc.) are treated as unsafe. This is a good thing. Unfortunately, for most running Rails 2 apps, automatically html-escaping our views without adjustments results in a hot steaming pile of onscreen view-source dung.

This is exactly what you see the first time you install the recommended rails_xss plugin to provide Rails 3 XSS protection for your Rails 2 app. Despite the obvious work entailed to get things back to normal, this is another good thing. Making the switch to Rails 3 overnight may not be feasible, but adding XSS protection ahead of time will help smooth the transition.

Except perhaps when your app is large. At the present count, we have over 1500 *.html.erb files in Weplay’s main repo. We simply can’t afford to devote the days to focus solely on fixes for excess html escaping in our templates. We need to be able to iterate on this change like everything else; incrementally instead of as an overhaul.

We came up with an alternative: we forked rails_xss to disable automatic html escaping by default while still providing the use of “html safe” strings and the Erubis ERB template engine. Developers can turn on html escaping locally using the autoescape block helper.

Everything within the helper is escaped. Anything outside the helper is not escaped. Nesting autoescapes is ok: everything inside the outermost invocation is escaped.

The autoescape helper allows us to apply the fixes on a view-by-view basis. We can devote most of our time to building features to help us make money and chip away at templates and partials as they pop up in our workflow. We even have a script to tell us how many files we still need to address.

The key to this change is manipulating the original rails_xss plugin behavior. Rails 3 helpers are composed of the String subclass SafeBuffer, which will escape concatenated content not already marked as “html safe”. Internally, calling .html_safe on a string simply sets a flag on the string object and this is flag is checked when adding to a string to SafeBuffer. By monkey-patching String, we disable the automatic escaping by always returning true for the html_safe? check. The autoescape flips on the flag, yields to the content block and flips the flag check back off afterwards. When autoescaping is already turned on, the block is returned as-is to allow nested autoescaping.

There are alternatives. We could all crash on the upgrade (no time!) or we could keep the fully “escaped” version of our app in a separate branch, chip away at it, merge occasionally (not fun!).

The opt-in approach works well for our agile mindset of fast incremental improvement and frequent deployment. I recommend using our fork if you have similar needs. It may even help with the heartburn.

Resources

• rails/rails_xss
• weplay/rails_xss
• Safe Buffer
• Nice explanation of XSS

As the game designer, I needed to build a game server that could easily handle multiple socket connections to player programs written by my classmates. Leveraging the event-driven nature of eventmachine, I was able to build a server that is fast, efficient and easy to maintain.

A commonly-cited drawback to event-driven programming is that it’s hard. For starters, we can easily wrap our heads around a procedural approach to a server connecting to a client:

1. Server waits for client 1
2. Client attempts to connect to server
3. Server accepts connection with client 1
4. Server tells client 1 to wait
5. Now server waits for client 2
6. Repeat as before and so on...

This pattern is familiar: like a cookbook recipe or instructions for building a model airplane, we are well-equpped to handle things one step at a time. With the event model, we think in terms of discrete behaviors instead of ordered steps. We need to identify important events and their responses:

• Whenever a socket connection is received, say hello
• Whenever a message through a socket connection, process it
• Whenever a socket connection disconnects, say goodbye

This model is common for front-end development (e.g. onclick, onmousever, etc.), but the notion of attaching events is not restricted to writing javascript for the DOM. Eventmachine provides some out-of-box hooks for adding behavior to expected events, like accepting socket connections. It also provides methods for defining and triggering our own custom behaviors.

Speaking of behavior: what about behavior-driven development with eventmachine? Absorbing the conceptual leap to programming to events is one challenge, and now, the mechanics of writing tests for them present another.

Let’s consider how we would test a client player connecting to my game server through a socket. I’ll build a class called Server which will be responsible for starting the eventmachine when its #start method is called. When a socket connection is made, I want to add the new player to a list of players kept on the server and send the message “READY” back to the client.

How would we write an integration test for this with rspec? The test would have to start an instance of my server class and separately initiate a client socket connection. Once the connection is made, then the test assertions can be run. After some digging around, I found some good examples to follow in the Ilya Grigorik’s eventmachine-backed websocket server, em-websocket. Here’s the basic approach:

An eventmachine process runs in a loop: once we start the loop, it will run forever until we trigger the event to stop the loop. All of our eventmachine code will happen in the event loop. We setup the loop by passing our code in a block to the run method:

So in our test, we’ll create the event loop with EM.run and pass a block with our test code in which we can start the game server and the socket connection. Once the assertions have been made, we’ll call EM.stop to end the event loop. Otherwise, it would run forever; we’d never get to the next test!

An eventmachine socket server can be initiated with the start_server method. We’ll give this method a host and port where it will listen for clients.

We’ll also use eventmachine’s connect method to make the client socket connection to the server. When a socket connection is made, eventmachine creates an instance of EventMachine::Connection which responds to certain methods representing different phases of the socket connection: after initialization, when a message is received, when the connection is broken, etc. EM.connect accepts a module or class inheriting from EventMachine::Connection which allows us to mix in our app or test logic to the connection.

To take the place of a player client in the test, I’ve borrowed FakeSocketClient (which subclasses EventMachine::Connection) from the em-websocket test suite and defined it in my spec_helper.rb. The fake client exposes attr_accessors onopen, omessage and onclose that we’ll treat like callbacks in the test.

We’ll assign a proc to the #onopen callback in FakeSocketClient. This proc will be triggered the first time the client socket receives a message. Since we want to the server to send a message when the connection is established, we expect this message to be “READY”. In addition, we’ll assert that there is one player added to the server’s list of players. Then we stop the event machine.

The key for making our rspec assertions: the socket connection between the server and client is accepted after the rest of the code in the EM.run block has been called. The instance FakeSocketClient receives #initialize and #post_init when EM.connect is called, but then the context returns to our test: we can now assign procs to FakeSocketClient’s onopen, onmessage and onclose callbacks as needed.

The #start method of our server is straightforward. It must start its own event loop and call EM.start_server previously discussed:

Other tests may include multiple client connections where we may need to assert that different messages like “YOUR TURN” and “WAIT YOUR TURN” are sent to the correct players.

For more reading on the subject of event programming, I recommend Dabek, et.al, Event-driven Programming for Robust Software.

At its core, the course is about solving puzzles. What is the shortest route that passes through all the cities (Traveling Salesman)? How many injured people around the city can you pick up in your ambulances in time (Ambulance Planning)? What strategy is best for winning more area than your opponents (Voronoi)? Who’s the best match across these 30 personality traits (Dating Game)?

Most of these problems fall into the category called NP-hard, which may be roughly defined as the class of problems that are at least as hard as those that can be solved in non-deterministic polynomial time. The more useful way for us to think about the general definition is to say that there is a verifiable best answer to the problem, but it would be very difficult, perhaps impossible, to design a general approach that would find this solution quickly.

Our goal for each problem then was to apply an appropriate set of heuristics: strategies to estimate a solution that would come as close as possible to the best solution in a reasonable amount of time. Two minutes in fact. Every week then, we had write a program to solve a NP-hard problem in competition with our classmates. If the program didn’t run, we lost. If it went over the two minute time limit, we lost.

Most problems involved interacting with a server via sockets. One student was chosen as the “architect” each week. Instead of solving the problem, the architect created an interface and protocol for communicating with the player clients, and a GUI to display the game progress and results. Extra pressure came with this role: if the server didn’t work or if player clients had difficulty connecting, the competition would be a failure. I chose to be the architect for the final game of the semester, Sudokill, a form of competitive Sudoku. I plan to write more about Sudokill in the near future.

By far, this course was the most difficult class I’ve taken in grad school, both conceptually and regarding the workload. At its peak, I spent 30-40 hours a week preparing for the Monday competition. This was often time spent in the early morning hours before and after work. By the end of the semester, fewer than 50% of students were completing their assignments on time. Despite the burden on my free time, this was also the most fun and rewarding class I’ve ever taken. It certainly helped that I was allowed to write my programs in Ruby; this is a rare opportunity in department where most of the faculty prefers Java and C++.

I’ve posted my class solutions in a single project on github:
https://github.com/rosskaff/heuristics

My code for the Sudokill game server:
https://github.com/rosskaff/sudokill

Check out the game too:
http://rosskaff.github.com/sudokill
Game rules

Several entries had already been submitted by the time I heard of the contest back in mid-July. I found games, like Lines, search engines, to-do lists, and so on. I considered implementing a Flickr slideshow that would make use of the contributions I’ve made to the supersized jQuery plugin at Weplay, but ultimately decided on a Keynote/Powerpoint alternative.

It’s called soapbox. The inspiration comes from showoff, which is backed by a server. soapbox provides an in-browser interface to create, preview and present slides with markdown or markup. Slides are auto-saved to local storage, if available, as json. I sent in my 10K Apart entry on the deadline day. You can also find the latest on github.

ROSS KAFFENBERGER #60
NEW YORK, NY
Age: 33 Gender: M

Distance Intermediate
Clock Time 02:24:27
Overall Place 82 / 360
Gender Place 69
Division Place 6 / 22
Swim 0:23:17
T1 0:01:42
Bike 1:12:41
T2 0:00:53
Run 0:45:51

Swim rank 67, pace 1:23 min/100 yds
Bike rank 126, pace 2:55 min/mi
Run rank 85, pace 7:23 min/mi