DM F#, Clojure, and Scala are all fairly new and popular languages this decade, the former two with striking resemblance to OCaml and Lisp, respectively, and the lattermost being more original in syntax. In what way do these languages represent forward thinking in language design, or how do they fail to build upon lessons learned in more venerable languages like Haskell?

KT It’s a common story in the world of software that time brings increasingly precise and accurate programs.  Anybody who grew up playing games on an Atari 2600 has witnessed this firsthand.  The image of a character has changed from one block, to five blocks, to fifty blocks, to (eventually) thousands of polygons.  By this modern analog of the Greeks’ method of exhaustion, Mario’s facial structure has become increasingly clear.  This inevitable progression, squeezed out between the increasing sophistication of programmers and the decreasing punishment of Moore’s Law, has primarily operated at three levels: the images produced by game programs, the logic of the game programs themselves, and finally the programming languages with which programs are produced.  It’s this last group that deserves special attention today.

The lambda calculus (and its myriad derivatives) exemplifies this progression at the level of programming languages.  In the broadest terms, you have the untyped lambda calculus at the least-defined end (which closely fits languages like Lisp, Scheme, Clojure, Ruby and Python), and the calculus of constructions at the most-defined end (which closely fits languages like Cayenne and Epigram).  With the least imposed structure, you can’t solve the halting problem, and with the most imposed structure you (trivially) can solve it.  With the language of the untyped lambda calculus, you get a blocky, imprecise image of what your program does, and in the calculus of constructions you get a crisp, precise image.

Languages like F#, Scala and Haskell each fall somewhere in between these two extremes.  None of them are precise enough to accept only halting programs (although Haskell is inching more and more toward a dependently-typed language every year).  Yet all of them are more precise than (say) Scheme, where fallible human convention alone determines whether or not the “+” function returns an integer or a chicken.  But even between these languages there is a gulf of expressiveness.  Where a language like C is monomorphically-typed (and an “int_list” must remain ever distinguished from a “char_list”), F# introduces principled polymorphic types (where, pun intended, you can have “’a list” for any type ‘a).  Beyond that, Haskell (and Scala) offer bounded polymorphism so that constraints can be imposed (or inferred) on any polymorphic type – you can correctly identify the “==” function as having type “Eq a => a -> a -> Bool” so that equality can be determined only on those types which have an equivalence relation, whereas F# has to make do with the imprecise claim that its “=” function can compare any types.

No modern programming language is perfect, but the problems that we face today and the history of our industry points the way toward ever more precise languages.  Logic, unreasonably effective in the field of computer science, has already set the ideal that we’re working toward.  Although today you might hear talk that referential transparency in functional languages makes a parallel map safe, tomorrow it will be that a type-level proof of associativity makes a parallel reduce safe.  Until then, it’s worth learning Haskell (or Scala if you must), where you can count the metaphorical fingers on Mario’s hands, though not yet the hairs of his moustache.

DM One might assume that increased “precision” in a language would come at the cost of increased complexity in concepts and/or syntax. In a research scenario, the ability to solve the halting problem certainly has its merits, but is that useful for modern commercial application development, and, if so, does it justify the steeper learning curve?

KT It’s absolutely true that languages that can express concepts more precisely also impose a burden on programmers, and that a major part of work in language design goes into making that burden as light as possible (hence type-inference, auto roll/unroll for recursive types, pack/unpack for existential types, etc).

However — to your point about the value of that increased precision – I would argue that it’s even more important in commercial application development than in academia.  For example, say you’ve just rolled out a new pricing server for a major client.  Does it halt?  There’s a lot more riding on that answer than you’re likely to find in academia.  And really, whether or not it halts is just one of the simplest questions you can ask.  What are its time/space characteristics?  Can it safely be run in parallel?  Is it monotonic?  In our business, these questions translate into dollars and reputation.  Frankly I think it’s amazing that we’ve managed to go on this long without formal verification.

DM Most dynamic languages have some air of functional programming to them, while still being fundamentally imperative. Even more rigorous imperative languages like C# are picking up on a more functional style of programming. The two languages you mentioned earlier, Cayenne and Epigram, are both functional languages. Are we moving toward a pure functional paradigm, or will there continue to be a need for imperative/functional hybrids?

KT What is a “functional language”?  If it’s a language with first-class functions, then C is a functional language.  If it’s a language that disallows hidden side-effects, then Haskell isn’t a functional language.  I think that, at least as far as discussing the design and development of programming languages is concerned, it’s well worth getting past that sort of “sales pitch”.

I believe that we’re moving toward programming languages that allow programmers to be increasingly precise about what their programs are supposed to do, up to the level of detail necessary.  The great thing about referential transparency is that it makes it very easy to reason about what a function does — it’s almost as easy as high school algebra.  However, if you’re not engaged in computational geometry, but rather need to transfer some files via FTP, there’s just no way around it.  You’ve got to do this, then this, then this, and you’ll need to switch to something more complicated, like Hoare logic, to reason about what your program is doing.  Even there, you have plenty of opportunity for precision — you expect to use hidden side-effects but only of a certain type (transfer files but don’t launch missiles).

But maybe the most important tool in programming is logic itself.  It’s a fundamental fact – well known in some circles as the “Curry-Howard isomorphism” – that programs and proofs are equivalent in a very subtle and profound way.  This fact has produced great wealth for CS researchers, who can take the results painstakingly derived by logicians 100 years ago and (almost mechanically) publish an outline of their computational analog.  Yet, although this amazing synthesis has taken place rivaling the unification of electricity and magnetism, most programmers in industry are scarcely aware of it.  It’s going to take some time.

I think there’s a good answer to your question in that correspondence between logic and programming.  The function, or “implication connective” (aka “->”), is an important tool and ought to feature in any modern language.  As well, there are other logical connectives that should be examined.  For example, conjunction is common (manifested as pair, tuple, or record types in a programming language), but disjunction (corresponding to variant types) is less common though no less important.  Negation (corresponding to continuations consuming the negated type), predicated quantified types, and so on.  The tools for building better software are there, but we need to work at recognizing them and putting them to good use.

Anybody interested in understanding these logical tools better should pick up a copy of Benjamin Pierce’s book Types and Programming Languages.

DM Many frameworks have a one or more DSLs to express things as mundane as configuration to tasks as complicated as UI layout, in an effort to be more express specific kinds of ideas more concisely. Do you see this as an expanding part of the strategy for language designers to increase expressiveness in code? Is it possible that what we consider “general purpose languages” today will become more focused on marshalling data from one DSL to another, or will DSLs continue to remain a more niche tool?

KT I guess it depends on what you mean by “DSL”.  Like you say, some people just have in mind some convenient shorthand serialization for data structures (I’ve heard some people refer to a text format for orders as a DSL, for example).  I’m sure that will always be around, and there’s nothing really profound about it.

On the other hand, by “DSL” you could mean some sub-Turing language with non-trivial semantics.  For example, context-free grammars or makefiles.  Modern programming languages, like Haskell, are often used to embed these “sub-languages” as combinator libraries (the “Composing Contracts” paper by Simon Peyton-Jones et al is a good example of this).  I think it’s likely that these will continue as valuable niche tools.  If you take monadic parser combinators for example, it’s very attractive the way that they fit together within the normal semantics of Haskell, however you’ve got to go through some severe mental gymnastics to determine for certain what the space/time characteristics of a given parser will be.  Contrast that with good old LALR(1) parsers, where if the LR table can be derived you know for certain what the space/time characteristics of your parser will be.

On the third hand, if a “DSL” is a data structure with semantics of any kind, your description of a future where programs are written as transformations between DSLs could reasonably describe the way that compilers are written today.  Generally a compiler is just a sequence of semantics-preserving transformations between data structures (up to assembly statements).  I happen to think that’s a great way to write software, so I hope it’s the case that it will become ever more successful.

• Level 0: 1 year
• Level 1: 2-5 years
• Level 2: 6-9 years
• Level 3: 10+ years

To be perfectly honest, I find the entire idea reprehensible. According to this matrix, nobody could be considered an “expert” programmer in C#, since the language has only been around since 2001. I’m not breaking the news to Anders. Ok, maybe that’s a bit of a “gotcha” exception, but I think the entire idea can’t hold water. The problem with the assertion is that it assumes that all years are equal in quality. There’s no comparison between a year in a challenging company on the cutting edge of your technology field working with the leaders in industry and a year making small changes to an enterprise CMS. No offense to the latter group, but it’s just the ugly truth.

What really bothers me is that this kind of fallacy isn’t limited to a few people, but, as I’m sure we all know, almost every employer has some kind of experience requirement in strict years. In fact, one employer I was looking at after graduation actually wanted candidates to have experience with C# since day 1, according to the advertisement. We, as an industry, have got to dispel the myth of causation (and even really correlation) between years experience and skill in development. Basing our recruitment standards on this primitive metric only encourages age-warfare, not unlike a number of discussions I’ve seen on Slashdot, with the younger programmers calling older ones “set in their ways,” and the older programmers calling younger guys “re-creators of the wheel in their inexperience.” It’s got to stop. We have to gather together and celebrate knowledge in our field and, in turn, work to better everyone in our technical community.

This is what I think makes Lab49 recruitment so effective: we don’t predicate someone’s candidacy on experience. Don’t have a degree in computer science or software engineering? Doesn’t matter. Don’t have ten years experience in software? So what? What matters is that you are skilled in your technology, be it C#, Flex, Java, or C++, and you’re someone who works well on a team. When you think about it, this is just good business. Who cares about someone’s pedigree? We’re in the business of creating effective software solutions, not showing off our employees’ education and past merits.

I think moving beyond simply removing the years experience criterion, résumés are becoming more and more deprecated. It’s not because people don’t have significant accomplishments that are noteworthy, but rather that people use ambiguous language, and even outright lies on their résumés in a desperate attempt to be hired for a job for which they’re not qualified, and for every gem in a stack of résumés, there’s a lot of rocks. This is where the community steps in. People involved in their local technology meet ups and people involved in online communities like StackOverflow show their skill on a regular basis, and it’s much harder to BS someone, especially if they start asking questions.

Over the next few years I suspect we’ll start seeing a paradigm shift in the way employers find talent. I won’t prognosticate that StackOverflow and its careers division will be the future, but I’m willing to be it’s going to have a hand in it, at least. The change in talent-hunting strategy, I think, is just one facet of a larger shift in our industry towards building communities. It’s already underway, so all we have to do now is embrace the change and jump in.

Part II. On the Phone

Sunday evening, January 18, I decided it might be a good idea to brush up on my .NET framework knowledge to prepare for my interview the next morning. Judging by the latter questions of Lab49’s “preliminary screening test,” these guys really didn’t mess around. I pulled off my bookshelf my trusty copy of CLR via C#, which is, in my opinion, the best book you can read if you really want to take your understanding of C# and .NET from “intermediate” to “expert”. C#  Developers: no excuses, read this book cover to cover. As it turns out, my interviewer, Nick, must be a fan of the same book. When he called me that Monday morning, after introducing himself, Nick threw me a couple softballs before turning up the heat. I was queried at length about generics, delegates, anonymous methods, and the garbage collector (among other things), all of which I was more than happy to explicate in the greatest of detail, having refreshed myself on their inner workings the night before. Nick’s attention then turned to the newer .NET 3.5 features, which I had been using for almost two years, and I was more than happy to talk about those, too. I must admit, he stumped me on a concept called “attached behaviors”. I was familiar with attached properties, but it wasn’t until recently that I’ve become fully aware of attached behaviors. I’ll have another article discussing what I learned in the future.

After Nick finished grilling me for information, I had my turn to ask him questions. I seem to remember having a list of things to talk about, but I was suffering from some strange variant of vertigo, so I went with my usual developer talking points. For the record, Nick is one of the nicest guys ever. As I would find out later, Lab49 is composed solely of superb people. You may be thinking that I’m generalizing or hyperbolizing, but in all seriousness, I have yet to find a single bad apple or even mildly distasteful person at Lab49. Every time I think I’ve found one, they prove me wrong. Even the Java guys are top notch, and that’s saying something. In any case, I finished the interview enjoying a discussion of the usual programmer minutiae, talking about podcasts and developer philosophy. I’m not sure if it’s normal for one to feel a sense of camaraderie with his interviewer, but I know I sure did.

Later that day, I received an email from James, a Recruitment Coordinator at Lab49, asking for a “telephone conversation” with Nemo, the Director of Recruitment. I figured it was one of those psychological profiles one of my friends had been subjected to in a recent interview. I don’t think I could have been more wrong. The next morning, Nemo called, introduced himself, asked for clarification on a few points of my résumé, and opened himself up for questions. I asked him the usual questions on how Lab49 was structured, the promotion strategy, and what Lab49 does in general.

Nemo explained to me that Lab49 is somewhat loosely structured, with no real middle management. As projects start and finish, you report to the project manager and engagement manager, but every project is custom-tailored to the clients’ needs. I pressed Nemo to explain how a successful company works without the infrastructure almost every company of equal size has. Nemo couldn’t really explain how it worked, but only that it did, and quite well. Some may say, “when the cat’s away,” but I might interject that maybe without the threat of imminent death, a mouse might be more free to do something more constructive with its time than cower for its life.

Coming from a consulting background, I could really tell Lab49 really is a consulting firm in the greatest sense of the term. The answer to everything is, “it depends,” and, “what the client needs,” which is clearly working out well for them, but can be frustrating trying to pin down something concrete on which to make a decision. In general, Labbers work on-site with the clients to best make use of human and information resources. In my opinion, there’s definitely a beneficial side effect to it: when Lab49 is working side-by-side with the client, the work has a face, rather than some unknown bunch of people dropping code in a folder every week or two.

Nemo continued by telling me that Lab49 has titles out of necessity, but they don’t play as much of a role as in other companies. I really appreciated that. Being a young guy, it’s typically difficult for me to get my ideas out in a space where I’ve got a title that’s easily negligible. I can say with authority that’s not been the case at Lab49. Every day I work with guys who have decades of experience on me, but are interested to hear what I have to say. It’s not about seniority; it’s about being the best at what you do and bringing new and innovative ideas to the table. For that very reason, I prefer to keep my title to myself. There’s no real company policy on the publicity of your title, but it’s not on my business card and I certainly wouldn’t wear it on my sleeve.

Nemo concluded the interview by saying that Daniel Chait, one of the founders of Lab49 would want to talk to me over Skype before an in-person interview. That was the first time anyone had said anything about an in-person interview, and while I had expected it at some point, I couldn’t help but be excited to see some light at the end of the tunnel. I confidently told Nemo that I was available for the remainder of the day, which might have caught him a bit off-guard. He said Daniel was a busy guy and he’d see when he was available. I was confirmed for later that afternoon by email not too long after hanging up.

I’m not sure if it was then or later that I started feeling extremely skeptical of this Lab49 place. I remember telling my friend, Chris, that, “I know Nirvana Corp burned down, but Lab49 makes me wonder,” quoting from an episode of Dilbert, The Animated Series.  I found it difficult to believe that such a place could exist, so little was my faith in humanity, let alone developers.

I then realized I was about to have my fourth communiqué with a company in three business days, whereas almost no Dallas-based company had insomuch as replied in a week.  These guys really don’t mess around, I remember thinking.

At 3:00 PM and after a few quirky Skype issues, I was on webcam with Daniel Chait, now fully appreciating the extra money I’d spent on a nice webcam. Chait went over my experience thoroughly, fully examining the roles I had played on each team. I’m not sure why, but at times I felt very intimidated. It’s perplexing because going over the conversation in my head Daniel wasn’t in the least bit condescending nor indirect in his questions. We discussed my work in WPF and LINQ and lambda expressions. Having taken Advanced Programming Languages and feeling like (but by no means) an expert in functional languages, I was glad to be on more solid ground. I had a compulsion to share with Daniel a couple extension methods I had written to succinctly state an equation in lambda. Unfortunately, since it was a compulsion I wasn’t prepared at all with the code snippet. I brought up my Machine Learning projects folder, which I knew it was in but couldn’t remember exactly which project I had written the extension for. Fumbling around, I must have spent a full five minutes of awkward silence finding the thing, which seemed anti-climactic for such a long wait, and even more so for me, since it felt like an eternity.

Afterward, I told my roommate that I wasn’t sure how that interview had gone. It was without a doubt by my estimation my weakest interview, and likewise the most important. I spent a little while going over the events, trying to find out why I felt I had done so poorly. All I can say is that I feel very confident of my technical skills and far less strongly about my experience, which was a major focus of Chait’s interview. That and that awkward silence.  I told myself that if they wanted a guy with more experience, that was a perfectly legitimate reason not to go forward, and I shouldn’t be worried about it. And if they rejected me outright for not having that bit of code on hand, they could go to Hell.

I was glad some of my friends wanted to go out that night so I could get my mind off the whole thing. When I got back, there was an email from James waiting for me, inviting me to New York for an in-person interview. I slept well that night.

The next day I made reservations airfare and lodging and ran both by James, as Lab49 kindly picked up the tab. It might be standard operating procedure for companies, but the fact is that they don’t have to do it, and I would have paid for the ticket myself if they hadn’t. Now all that was left to do was wait out the week and familiarize myself as best I could with my travel plans.

Clearly this company had their act together and was vigorously pursuing this discovery period. I was also reacquainting myself with the ever-greater possibility of leaving everything for New York. My father was supportive, and because he’s not usually too keen on my adventures, I took it for a good sign. That being the case, I started warming my close family and friends up to the idea that I might end up in New York. Most of them were surprised, marveling at the apparent spontaneity and haste in the whole process. Only months before had I told them I’d be staying in Dallas for the “foreseeable future” as a consultant writing software for my own company.

Who knew the foreseeable future was so short?

Part I. The Call

I had been consulting for right about a year and a half, and things were looking pretty good. Chris and I, under the company name of Gestault Solutions, had managed to architect and implement a complete resource scheduling system for outpatient healthcare centers. We were not without the help and business guidance of Randall, who is perhaps one of the most intelligent and honorable businessmen I will ever meet.  I’ll be the first to admit, it was an incredible undertaking for such a small team in such a short time, and I while I may wince at a few spots in the codebase, I’m proud to have worked with Chris and Randall on it.

Of course, every company has problems, and while ours weren’t so much in the codebase, but in those pesky people we call “clients,” who seem to be the most disorganized and incoherent bunch of people under a roof. I really hate business politics. Our main client, to whom we were very dependent upon for revenue decided that they wanted to fork our legacy product, iOrder, and use their staff to continue incremental feature improvements on it. We had concluded several months earlier that while iOrder was doing very well for when it was written and the circumstances thereof, that it was outdated and had several architectural flaws that hampered further development. Fortunately, Chris and I were both very adept at the suite of new .NET technologies, notably WPF and very quickly WCF, so we were able to leverage both of those frameworks to get well-designed software out the door orders of magnitude faster than teams using older technologies.

Our client’s team was expecting to make a release in September of 2009 with a pittance of feature improvements. Since we were brought on to help their team transition, we had a good idea of how things were progressing, and it was our professional opinions at the time that their chances of success were slim to none, based on the skill sets of their developers and architectural choices. It’s worth pointing out that these guys were accustomed to maintaining software, making tweaks and small changes and this was new development, which requires a whole different set of skills. It may not surprise you then that as of this writing, they still haven’t finished their handful of upgrades.

Around the same time as the client’s stated release date, we were testing our completely new WPF user interface with a good bit of success. There were some initial bugs, but that’s to be expected. Despite having blown their own release date and given the opportunity to ditch their ailing project and purchase our finished one, our client (if you could call them that at this point) decided to persist with their team with a slightly revised schedule. I’m not sure how many deadlines they had, but they are now aiming for a June 2010 release. From what I’ve seen, a pig with a model rocket strapped to its back has a better chance of getting off the ground.

We continued working on our project, albeit at a lower rate through the end of 2009 and beginning of 2010, but it was looking like we were not going to pick up again anytime soon. Randall had found a second client to bring onboard, but they weren’t in a hurry to do anything either. I had just graduated in December and was looking to kick off my career, but every week things became a bit grimmer, despite Randall’s best efforts to show our main client reason.

I wasn’t really worried until Chris started interviewing. That was when I got my résumé in order and started making phone calls. The job market wasn’t looking very interesting to me; there was a lot of web work, internal management systems, so forth and so on, but very little WPF work or companies on the cutting edge looking for employees. I was also looking for a senior position, which may seem a bit audacious for my age and credentials, but I was more than willing to prove to anybody that I wasn’t wasting their time with my job application. A recruiter called me and introduced me to a particular FedEx/Kinkos project designing some kiosk interface with a bunch of contracted developers. I was pretty skeptical, since without strong project leadership, success becomes a fleeting goal, and I’d hate to have my name pinned to a failed project, even if it wasn’t my fault.

On Friday, 5 January, I was called by a guy who introduced himself as Ryan Elberg from Lab49, a software consulting company. He noted, “I saw your résumé on Dice and that you were available for relocation.”

“Depends on the job,” I quickly replied. “Where are you located?”

“New York. Are you interested?”

I had marked the “Available for Relocation” checkbox on Dice, the monster.com of IT, in case a company in Houston, Austin, or San Antonio was looking for a programmer.

Not really knowing what to say, I went with my previous remark, “Depends on the job. Which part of New York?” It could have been Albany, for all I knew.

“Downtown New York City.”

At this point I was just stalling for time. I had no real intention of moving to New York, but there are a lot of premier software companies in NYC, so why waste a good job opportunity? I quickly decided that I’d go through the interview process, which would surely include a face-to-face interview, at which time I could decide whether I wanted to be there or not.

Ryan asked me when I would be available for a quick pre-screening test. I told him I was ready immediately, so he commenced with a little multiple choice test over the phone. Apparently, I did quite well, so he set up another phone interview with a developer named Nick for the following Monday.

After I hung up, I couldn’t help but feel a bit exhilarated. It’s not everyday people quiz me on CLR garbage collection and such. At the same time, I was filled with dread toward to idea of leaving my friends and family behind in my beloved Texas, a bastion of freedom and conservativism, for New York, a place with neither family nor friends, except my Uncle far upstate in Rome. And New York City was recently in the news for Mayor Bloomberg’s thoughts on how someone with a concealed handgun license could shoot up a movie theater. Not exactly the kind of rhetoric I was accustomed to in Texas.

I decided to procrastinate on figuring out what I wanted to do. Being fresh out of college, I was an expert at putting things off until later. Maybe something would happen along the interview process where I could find an out and not end up moving. It’d be interesting to say that someone called me from New York thinking I’d be great for their team, but I turned them down because they didn’t have it together.

I was in for a surprise.

Continued in Part II.