Blog

Jan 29 2013

Comments

Alex Warr - Podcast Episode 025

cover art

When potential customers decide that they might want to work with us, their first point of contact is our sales department. It's there that they get their first impression of what sort of company we are. So it has been clear to me for quite some time that we needed to have Alex Warr, the head of our sales team, on the show to talk about what sales is like at Relevance. Along the way, we also managed to talk about babysitting, gaming, law school, and a few other things besides. Enjoy the episode!

Read More »

Jan 28 2013

Comments

What We Learned, and the End of the Magic

When we started the Word Magic project, there were some things we wanted to learn, and some things we wanted to confirm. This post is about those things.

First, we confirmed that the Big Data Reference Model holds true. We tried to operationalize analysis before we had really finalized the kind of information we wanted to produce. (Most specifically, how the numbers behind the "Word Usage Through the Decades" module would work.) In terms of the Reference Model, we were skipping the first two learning loops. Why would we do that? The same reason everyone else does. We thought we knew what we wanted.

Our initial mockups looked great, but they were foiled by the real data. The algorithm was to divide a word's occurrence in a decade by the occurrence of all words in that decade. That way, we expected to normalize out the general increase in written language since the 1600s. But, since we put both a phrase and the individual words on the same graph, we normalized to the largest occurrence of them. Sounds good, but in practice, the bubbles were all but invisible except for the most common word in the phrase. The most common word dominated the graph and caused the others to be invisible.

We eventually tried two visualizations and three algorithms before we got one that was suitable for launch. There are some different calculations that might have worked out well, but we couldn't try them. Why not? Because we jumped straight to operationalizing. Every time we changed the calculation in map/reduce, we had to spend multiple days reprocessing 400 years of humanity's entire printed output. We should have followed our own advice and done smaller samples in a quicker toolset.

The second thing we confirmed is that JRuby on Hadoop is a viable way to write map/reduce jobs. It's a thin wrapper on top of the Java API, but still an improvement.

Stuart Sierra's clojure-hadoop library also worked very well. Because this was a "20% project" for most of its life, we ended up with pieces written in JRuby, Clojure, and Java. The polyglot nature didn't really present a problem, because each piece was its own data flow.

On the deployment front, we used Pallet. It worked very well for deploying whole topologies of machines. We built several Hadoop + HBase + MapReduce clusters with it.

Finally, we confirmed that the Hadoop and HBase APIs are cumbersome, poorly documented, and frequently misleading. The processing model is expensive, and the query model is user-hostile. (We aren't the only ones to notice this.) Moving between our daily world of sharp tools like Ruby on Rails, Clojure, ClojureScript and Datomic and into the Hadoop world definitely felt like a step back in time.

Now that we've learned what we needed to learn from Word Magic, we have shut it down. Hadoop is great for scaling up with lots of servers, but it doesn't scale down well at all.

Jan 15 2013

Comments

i-Human - Podcast Episode 024

cover art

At Relevance, our mission is to help people solve hard problems through software. Most of the time, we do that through consulting. So it seemed well past time that we had one of our consulting clients on the show. And in my opinion we could not have made a better choice than Anne Tweet and Jay Patel of i-Human. We got a chance to talk not only about their experience working with us, but also about the really interesting software that they're building. I had a great time talking with them - I hope you, too, will enjoy this episode!

Read More »

Jan 14 2013

Comments

Taming side-effects with mutability

Clojure prefers immutable values and referentially transparent functions. But it also works well with the Java ecosystem, where everything is an object... mostly mutable.

I was recently dealing with ASM, the bytecode manipulation library. Most of the API is based on the Visitor pattern. To read a class from bytecode, you construct a ClassReader object from an array of bytes or an input stream, then tell it to accept an object that implements the ClassVisitor interface:

public interface ClassVisitor{
  void visit(int version, int access, String name, String signature, 
      String superName, String[] interfaces);
  void visitSource(String source, String debug);
  void visitOuterClass(String owner, String name, String desc);
  AnnotationVisitor visitAnnotation(String desc, boolean visible);
  void visitAttribute(Attribute attr);
  void visitInnerClass(String name, String outerName, 
      String innerName, int access);
  FieldVisitor visitField(int access, String name, String desc, 
      String signature, Object value);
  MethodVisitor visitMethod(int access, String name, String desc, 
      String signature, String[] exceptions);
  void visitEnd();
}

Well, now we see a bit of a problem. The ClassReader gets to call these methods, which mostly return void. When a ClassVisitor does return an object, it returns yet another visitor that the ClassReader calls to descend into members of the class. The methods of the visitor get access to information about the class, but how do we get it out?

In other words, how do you use an API that is 100% about side-effects and mutability inside a language that favors immutable values and pure functions?

One solution is to use mutability to contain the side-effects within a function, such that the function is referentially transparent from the outside. In so doing, we tame the side-effects and put a boundary around them:

(defn analyze [c]
  (let [classinfo (atom {})
        v         (reify ClassVisitor
                    (visit [this version access name sig supername interfaces]
                      (swap! classinfo assoc 
                             :static?    (static? access)
                             :interface? (interface? access)
                             :final?     (final? access)
                             :classname  name 
                             :super      supername))
                    (visitSource [this source debug])
                    (visitOuterClass [this owner name desc])
                    (visitAnnotation [this desc visible])
                    (visitAttribute [this attr])
                    (visitInnerClass [this name outername innername access])
                    (visitField [this access name desc sig value]
                      (swap! classinfo update-in [:members] conj 
                             {:type       :field
                              :visibility (visibility access)
                              :static?    (static? access)
                              :final?     (final? access)
                              :name       name})
                      nil)
                    (visitMethod [this access name desc sig exceptions]
                      (swap! classinfo update-in [:members] conj 
                             {:type       :method
                              :visibility (visibility access)
                              :static?    (static? access)
                              :final?     (final? access)
                              :abstract?  (abstract? access)
                              :name       name})
                      nil)
                    (visitEnd [this]))]
    (.accept c v 0)
    @classinfo))

In the let form at the beginning, I create a new atom that holds an empty map. This will be my mutable state for this function. Then I create an instance of ClassVisitor, where most of the methods do nothing and return nil. Only visitClass, visitField, and visitMethod are interesting to me. In those, I mutate the atom to attach the info that ClassReader passed to the visitor.

After the setup in the let, all that is left is to call ClassReader.accept with the new visitor. At that point the ClassReader does its bytecode interpretation and makes all its calls into the visitor. Finally, I get the resulting value out of the classinfo atom, which now has a nice, immutable Clojure-friendly map of information collected from that class file.

This technique is similar to using transients to build collections: allow mutability within a function, but don't let it escape. Transients do it for speed. The ClassVisitor does it to bridge to an object-oriented API.

Jan 03 2013

Comments

Justin Gehtland - Podcast Episode 023

cover art

Way back at the end of 2011, I had this idea that maybe I could record some conversations with various Relevancers and throw together a podcast. Our very first guest was Justin Gehtland, so as the end of the year and the anniversary of the show approached, I thought it would be only too appropriate to have Justin on again. Besides, he's a great guest, and we were clearly overdue to have him on the show again.

As is traditional for this time of year, we spent a good chunk of our conversation talking about the year gone by (verdict: it was crazy) and the year to come (verdict: it's super exciting).

I hope you enjoy this episode!

Read More »

Jan 02 2013

Comments

Where to Find Relevancers: January Edition

Want to meet a Relevancer in person? Here's where you can find us during the month of January:

Sandusky, OH 1/8-1/11
CodeMash
Speaking: Jen Myers Talk: Straight-Up Design: Simple Ways for Devs to Make Apps Look and Work Better

Durham, NC 1/9-1/30
Girl Develop IT - RDU Intro to HTML & CSS @ Relevance HQ
Attending: Lynn Grogan

Richmond, VA 1/15
Richmond Ruby User Group
Speaking: Russ Olsen

Denver, CO 1/17
Den of Clojure Meetup Group
Attending: Timothy Baldridge

Durham, NC 1/24
Triangle Clojure User Group Meetup @ Relevance HQ

Dec 24 2012

Comments

The Real Secret Behind Project Failure

To the owner of a failed software project, there is little consolation that roughly forty to seventy out of every hundred other projects have also failed to meet their goals.

This sad math reflects a simple truth - delivering software solutions to hard business problems is, well... hard. Teams lacking sufficient technical skills are obviously doomed from the start. But a respectable lineup of developers merely gets you in the game. Projects making this first cut now must endure an onslaught of other challenges, from ambiguous requirements and unclear priorities to unrealistic schedules, poor communication, insufficient planning, and unmanaged risks. In other words, as a product owner embarking on a new assignment, the biggest threats to your success are not technical problems at all. At Relevance, we understand that software development is mostly a people problem.

This is why aside from employing an amazing group of developers, designers, and architects, most Relevance projects receive the direct attention of a Relevance coach to minimize the non-technical risks to your project, and to help great people accomplish great things.

So what exactly does a coach at Relevance do? A partial answer can be found in an episode of the Relevance podcast, but the conversation is sporadically interrupted by loud monkey noises (oddly not present at the time of the original recording). In true agile fashion we shall adapt and repeat, and our answer will focus on outcomes rather than activities. When clients ask us "What is the role of a coach at Relevance," we've found what they're really asking is "How will my project benefit by having a Relevance coach on the team?"

Read More »

Dec 12 2012

Comments

Timothy Baldridge - Podcast Episode 022

cover art

It may sound trite to say so, but Relevance is the awesome place it is because of the people that work here. So I thought it would be interesting to talk to our newest hire, Timothy Baldridge, about what it's like to get hired. He had a lot to say on the subject, and I found it pretty interesting. Just as interesting was finding out more about Clojure-Py, Timothy's implementation of Clojure on the Python runtime.

So please download the episode and enjoy!

Read More »

Dec 05 2012

Comments

Jason Rudolph - Podcast Episode 021

cover art

After Stu and Justin founded Relevance and started to grow it into the company it is today, one of the first things that they did was to hire Jason Rudolph. For that reason alone, I have long wanted to have him on the podcast. As it turned out, he had his premiere on the show with the Rails Rumble episode a few weeks ago. But way back in August I had already finally taken the time to sit down with him and chat a bit. Things being what they are, it has taken me until now to finally edit and publish that first episode.

Anyway, we got a chance to talk about his evolving role at Relevance, his recent experiences with ClojureScript, and his life as a productivity nerd. It was fun! Download the episode here. Enjoy, and thanks for listening!

Read More »

Dec 03 2012

Comments

Popular Tags