It’s not a new version of Visual Studio, but they’ve switched over to it full time for six months now, they’ve finally shipped another build and they’re started developing new language features because everything’s stable since they ran out of old ones and let everything settle. If Roslyn has an unofficial 1.0, it happened yesterday.
Roslyn was the codename of the effort to rebuild the C# and Visual Basic.NET compilers in their own languages, but also to do it in a modern way. These compilers expose services that are appropriate to the stages of compilation and allow the information that the compiler builds up not only not to go to waste but to be readily accessed by other programs. Instead of sitting on the knowledge, sharing it.
So it’s only fitting that Roslyn went open source under the Apache 2.0 license yesterday; that notes from the language design meetings, at least partially, are being catalogued on the Roslyn forum (which has already sparked a discussion about attribute-based metaprogramming where the tender of Scala’s macros jumped in for some interesting discussion) and that the first thing Anders Hejlsberg did on stage at Build was to fork Roslyn and implement a language change.
There are so many bottoms to this. Languages are changing constantly. Microsoft is learning to listen more to its developers. A language has never been owned by its creator, but even less so now. Considering what they can do when they don’t have the adequate tools, empowering everyone outside the moat (and from all I’m hearing, inside too) will net rewards we can’t even imagine.
There are open languages now, there’s been metaprogramming for ages and experts are butting heads every day. It’s not new. (Although having a service-oriented compiler architecture is still less common than it should be.) What’s new is that even the company that’s going broke trying to sell you $13,399 copies of Visual Studio is now populated mostly with people who see this as the new normal and the desireable frontier where the most good will happen.
By “at it”, I mean tearing up fundamental assumptions about how computation can be used to create art or turn data into visualization.
By “again”, I mean that he’s finally put up videos of some of those talks. Two videos already, and the third one this Tuesday; all basically within the month of May.
Some of you people will demand of me: “this is a programming weblog, the only sort of installation I’m interested in is of apps, and as for visualizations I’m barely into syntax highlighting”. That’s a strawman for all I know. But if you do feel this right now, treasure this feeling. Write down exactly what you feel and set a calendar event for spring 2023. You will have been part of a shift in thinking that comes along once or twice in a century, and you will look back at this feeling like a teenager at a crib, wondering how something so constricting could ever have been acceptable.
The two videos that are out already is the story of writing up long-winded instructions with exact timing, handing them off to a mechanical food maker with so many foots and levers and hoping that something edible will come out this time. It’s not actually about that, but it’s as if we did all that before, we still do and Bret Victor is showing us for the first time how to cook.
I’m looking forward to Tuesday.
Sidecar interface files are available for jQuery, jQuery UI, node.js, the DOM and more. CommonJS modules and AMD modules both supported for consumption and production. The compiler is self-hosted and fully open source.
Thought experiment. Imagine if you bought a new microwave, took it out of the box, and found a panel of unlabeled buttons. Imagine if the microwave encouraged you to randomly hit buttons until you figured out what they did. Now, imagine if your cookbook advised you that randomly hitting unlabeled buttons was how you learn cooking.
Also, a new take on an old favorite (paging Nathan):
In his influental essay No Silver Bullet, Fred Brooks makes the case that software is inherently “invisible and unvisualizable”, and points out the universal failure of so-called “visual programming” environments. I don’t fault Fred Brooks for his mistake — visual programming is indeed worthless. But that’s because it visualizes the wrong thing.
Traditional visual environments visualize the code. They visualize static structure. But that’s not what we need to understand. We need to understand what the code is doing.
Readers are advised that they will henceforth consider their tools puny and broken. (Maybe it’ll even get a few vim- or emacs-heads to question whether a pure textmode editor could hack it if the stakes were raised. The few of them who will accept the idea that a couple of men didn’t get everything right in the 70s, that is.)
(Some of you may remember previous ruminations on an “xlang” language. I don’t consider this to be it; “xlang” was my label for Apple’s side-by-side-cessor to Objective-C that would be compatible in the same way eero’s compatible, but that would be released by them, have full support throughout all the tools and probably introduce at least one new paradigm in the way that, say, blocks was a new paradigm. eero is just about as close as anyone outside of Apple can get to it, though.)
GitHub releases ReactiveCocoa, the “magic between GitHub for Mac” and “essentially an Objective-C version of .NET’s Reactive Extensions (Rx)”.
Composable operations on futures: are they here to stay?
Five and a half years ago, Joel Spolsky announced that they had indeed graduated writing FogBugz from VBScript to an in-house language defined as a superset of VBScript called Wasabi, reimplemented using the semantics that suited them, and that they did this instead of porting the code to, say, Ruby. (The day after, the rest of the Internet announced that Joel Spolsky had indeed graduated from occasionally correct to stark raving mad. That link is from Jeff Atwood, who would naturally go on to start a company with Joel.)
Two years ago, Facebook announced that they had indeed been working on switching over from whatever standard-ish (but likely tuned) PHP machinery they were using to their own HipHop cross-compiler, which would spit out C++ that would then be compiled and work faster.
Joel may well have been right in that Wasabi, having been written, could smooth the transition to a non-caveman language by introducing smarter features, not breaking the current code base and finally allowing a high-fidelity cross-compile to another language where it may be kept from there on out. (I have no idea how they got there, but I know that they are now on C# and ASP.NET. And let’s ignore for a minute the labor that went into maintaining the compiler, training the programmers and so on.)
But what to make of HipHop? Google has never had problems with scaling because it started out focused on solving a scaling problem and the culture never left them. For the first few years of Google’s existence, being able to fit, index and work over many more pages than any other search engine was part of Google’s advantage.
Facebook might not be able to rewrite everything over the weekend, but they pride themselves on rapid iteration. Surely they could figure out how to map everything onto a new architecture with different tools (possibly by inventing a new language, maybe channeling people like Andrei Alexandrescu) and perform the migration over a non-disastrous period of time.
Eevee is right: PHP is a monster of a bad language. (For the rest of my life, barring something even better, I will carry around that link to use when people ask why I hate PHP. A third of the text could be slander and a third out of date and it would still be enough to keep me away. It may run the Internet, but it does so in spite of itself, and not because of itself.) It doesn’t matter to Facebook’s customers that Facebook is written in a sub-standard language, but it matters to Facebook that Facebook is written in a sub-standard language because it gates their productivity and multiplies their maintenance and support.
It seems that half of my posts here end with hopes that there will be a new perfect language which will solve everything. While I would love that, I don’t think it’s coming.
What I do see are opportunities. Rather, thanks to Bret Victor, I see pain and suffering where needs are not being met. There’s a place for a few languages to be excellent in their field. (The most vivid image is of a proto-C# to replace C++ that would be efficient and not assume you were the world’s foremost expert on memory management, but actively take care of it itself. I don’t just mean garbage collection (or maybe not at all), but a virtual machine or JIT that would recognize patterns and say “hey, I know how to implement this pattern super effectively!”, beyond just solving general cases.)
When companies do things like Wasabi or HipHop, it’s a sign of something more. It is the first inkling of a desire to not accept the limitations of what they have before them. At their best, it’s also a sign that they don’t accept the limitations of any other method; at their worst, it means that they are so set in their ways or so cheap that seeking out something else doesn’t turn into an option no matter what. Sometimes, you can’t tell except by doing it and looking back.
This is not a scale from point Ew to point Ah, and the pot of gold at the end is not “inventing a new language”. But being willing to transcend every readily available option is the spark that is necessary to change the world, improve the state of the art and make life better. I wish we’d do it more often.
Around ten years ago, development started on X# at Microsoft Research, later renamed Cω (C-omega). Cω attempted to unify data stores like XML documents and databases with the C# language.
Cω eventually became the LINQ project, standing for Language Integrated Query. Below only .NET itself, the new Metro-style/WinRT infrastructure and some early Internet work, it may be the broadest-reaching endeavor to ever be undertaken within Microsoft. LINQ became a way to write SQL-like queries, translated by a series of paper-thin syntactic transformations to extension methods/”query operators”, library methods or your own for producing or manipulating a sequence. LINQ on its lonesome forced VB.NET and C# to adopt a slew of features to make sure that queries could be written due to the relatively meager standards of version 2 of those languages.
But it didn’t end at letting C# programmers stop hiding their head in their lap whenever they were asked why Map was missing from a standard library with thousands of classes. Part of the supporting infrastructure was a way to turn the lambdas used for implementing the higher-order function/”query operator” input into expression trees, a language neutral enough sort of syntax tree, and the concept of a query provider. A query provider could, given that the expression tree versions of the queries were used, recompile or reinterpret the query into another query format. Most commonly, this is used to translate the query to various SQL dialects, but there are also query providers for XML and LDAP.
The first SQL query provider came from LINQ to SQL, a trailblazing ORM originally meant as a proof of concept and which only worked for Microsoft SQL Server. It took two years for Microsoft to deliver Entity Framework, “EF”, the real ORM which also works for every database with the correct “provider”, and one more major version for Entity Framework to grow into its role. EF is now updated a lot more frequently and slowly pleases more and more developers.
Which brings us to today. LINQ won’t do much for key-value databases that just model a dictionary (even if, and especially if, smart such databases, like Redis, take the opportunity to model useful operations on the values like set sorting or atomic increments). And while it works for traditional SQL databases, it hasn’t eliminated the difference between objects and database concepts, it has in a sense highlighted the differences that exist even more. I may not be shuttling data on or off
DataTables, but I am more aware than ever about what’s happening where. I have to be slower about adopting new features in my favorite database to make sure the entire chain supports it, and some databases might not have providers or might have buggy providers. In some ways, LINQ and Entity Framework have driven me into the uncanny valley of database connectivity.
This isn’t purely a problem of using “Microsoft’s solutions”. To a large degree, it’s the cost of picking this sort of approach. Traveling less audaciously might be calmer and more consistent. But the box has been opened and the bobcat is clawing wildly at the portrait of gramps on the mantlepiece. You won’t get the painting back, but you’re going to anger a whole lot of
bob-er, developers if you try to put them back in their boxes. LINQ-like principles have been able to be replicated in Scala, gleefully (to some) without having to add language features. Rails’ ActiveRecord, since a while back, uses an SQL builder approach of constructing the query and translating it into SQL at the last minute for the best performance. The principle is sound, and although it’s not perfect, it also hasn’t been carried to its logical conclusion.
What’s left seems like low-hanging fruit. I have heard nearly nothing about it, but I’ll be damned if twenty people aren’t sitting around working on it somewhere right now: There needs to be an object/node/graph database built explicitly on LINQ-like principles and in a way as to optimize for LINQ-based usage patterns. The data conforms to object patterns instead of its own designs. The kind of queries you send it informs a dynamic query analyzer, rebuilding and maintaining indexes for optimal access. You don’t have to worry, really, about primary keys, but more about references to other things.
Yeah, it’ll take a new approach and we’ll have to toss what we have. Good. It doesn’t look to me as if we’re in the best state we could be, nor as if there’s been a ton of recent research in this direction. There have been object databases, but only to the point where there’s barely a new one every couple of years and they ironically don’t smell of being open to fresh approaches. And Microsoft Research (again) is working on Trinity, a distributed graph database, but it’s not externally available and seems disproportionally involved with serialization to and from binary messages (and that’s ignoring the references to the cast of Friends in the manual).
Fundamentally, this isn’t that involved with LINQ. But LINQ is a nice handle for what this new database should be – assuming code access instead of database operator access, eliminating the divide between the database and the accessing system and finally giving us a new set of tradeoffs for a new decade.
Mono 2.11 ships, including C# 5.0
async, compiler-as-a-service (different in scope and approach from Roslyn) and partial significant .NET 4.5 API before Microsoft’s stable versions of any of those things.
Say what you will of Miguel de Icaza, but I think he gets things done.
Microsoft hosts Lang.NEXT early April. Features, among other things, the minds behind Scala, D, Dart and C#. Listen to the inspiring, mind-opening rhetoric from captains of the industry; to fresh, mould-breaking approaches from bright researchers; or to Herb Sutter.