I’ve been absorbed lately with the nagging question that keeps coming up every time I solve what I thought was going to be a simple problem – “Are we the only ones doing this?” This is followed by the more important question – “How is everyone else doing this?” At NetQuarry, we sell, effectively, answers to these questions. The problem, and the thing that always suprises me is the fact that we have to do much of what we have to do to produce our product. The complexity of enterprise application development is startling. You basically need to be an expert on about 10 things just to get started.

I’ve been personally involved in an enterprise architecture project for a well funded valley company (it was J2EE, to be fair) that had 100 people (including QA, Docs, and a bunch of managers) working for 2 years to implement an enterprise application architecture. We spent nearly 40 million dollars (seriously) on development. I’ve spoken to CTOs at software companies (ISVs) who are solving this and have arrived at the conclusion that nobody knows exactly where to start.

Most people in my position would roughly agree with me, at least in principle, but most would have a hard time citing real examples of why this is so hard. So, here are some examples:

Simple example first – permissions.

When you build an application of any reasonable complexity, you will have to architect and implement a way to authenticate users, retrieve a set of profiles (groups), and restrict access to parts of an application based on who the user is and which profiles have been assigned to them. How could you accomplish this? There are a ton of ideas out there. There is an idea of a database directory that .NET will build for you in SQL Server. There is a decent way to hook in Microsoft Passport authentication. There is a user/password/reset password control. There is the “forms based authentication” that takes a real time investment to understand and once you do you wish you hadn’t. What doesn’t exist anywhere is a reasonable way to consume the information you get from your authenticated users.

What also doesn’t exist (and this is the important bit) is a way for you to declaratively hide a column in a grid based on which user just logged in. Hold on, I’m going to explain.

Say I have a column in my DataView grid (say a salary column) that I only want users with the Administration Profile to see. Further, I want to define this by checking a little box somewhere that says “Hide from general users, please.” Then, I want that information stored in some persistent format that allows me to check it into source control. Asking too much? You might say this is easy. All you have to do is… build a file that holds the data, or put the data in a database (this sort of discussion ends relationships), write a little tool that edits the file or just screw with it yourself, store this file in source code, add it to your installation script, build a tool to change it inside your application (which you need to secure as well), copy it to the server before you go live, … Crazy.

Another example  – filtering

The .NET Framework and Visual Studio, along with the competition (J2EE, and the slew of tools available here) has a way to take data out of a database and present it in a grid. Generally there is a way to handle sorting, paging, hiding columns, and, of course, applying some sort of theme. However, what if you want to filter the list in some way? Simple, right? All you do is build another set of controls on the form for each column that you want to use in your filter. Some of the columns are foreign keys to other tables, so you’ll want to build some sort of interface to deal with these. Many of your foreign keys are really just lookup values, so you’ll want a way to build a DropDownList control with these values in it. Next, you’ll have to take the result of a postback on this form, determine if it has to do with your filter controls, and construct some sort of SQL filter for your database and re-display your list. If your users want to be able to filter on multiple values, well, then, its a bit harder. You’ll need either a new interface, or you’ll have to handle filters like: “BETWEEN x AND y” or “This Month” or “>Today” or “=FRED” or “Starts With Fred” and likely a bunch of others.

The kicker – you have to do this with every grid. You’ll probably get clever and build a base class that reads information from the DataSource and builds some filtering capabilities. Once you’ve done this, you’ll think its pretty easy. Honestly, though, a full featured grid view with good filtering (forget about column wise permissions) is a bunch of work. Even if you are pretty good at this you’ll still have the problem of managing the thousands of foreign key lists. (Much more on this idea here.) And, each one is just a little different. So, you probably end up with a bunch of pages all with slightly different code in them to change the properties of the particular page you are working with. At least you can check these into source control.

Meaning?

Enterprise application development is hard, really hard. Any way you look at it, you absolutely must solve most of these problems, even if you solve them poorly. Generally this is done by giving your Highly Paid talent the title of Architect and asking them to guide you through this process. If you are smart you’ll get two of these smart folks, because one smart guy alone will always come up with dumb ideas. If he is working by himself then you’ll usually end up implementing some of these ideas. Loosely quoting Brian DePalma – “Every time you think up a perfect architecture there are 50 ways to screw it up. If you think of 25 of them, you’re a genius.”

The solution always looks about like this:

1. Hire 1 or 2 expensive people and let them think for 3-6 months.
2. Add a productive resource and start writing a low-level database access class library that wraps ADO.NET. or JDBC. This puts you 6-9 months into the project.
3. Add another productive resource and build a reasonable grid, detail view, and a simple navigation metaphor. You are at least 9 months into the task now.
4. Frantically add a bunch of not so productive (read, cheap) resources and tell them you are behind. Sometimes you send your application to India, China, or Iowa at this point.
5. Scramble for another 6 months or so until you begin slashing features as fast as you can.
6. Release your first version 18 months from the start of your project to resounding cries of disappointment and astonishment from your user base.

This isn’t just me talking. There are well known research reports from respected analysts that assure us that more enterprise projects fail than don’t. Below, from Software Magazine, a blurb on the Standish CHAOS report:

“The 10th edition of the annual CHAOS report from The Standish Group, which researches the reasons for IT project failure in the United States, indicates that project success rates have increased to 34 percent of all projects. That’s more than a 100-percent improvement from the success rate found in the first study in 1994.”

So, we’re getting better. Only 66% fail. I’m not suggesting that poor architecture implementation is the only reason projects fail. I’m betting that it isn’t even the leading direct cause. But I do know who gets blamed when the project doesn’t go correctly – the Highly Paid developers. And, I believe it isn’t necessary.

Why hasn’t this been solved?

I believe that eventually Microsoft will introduce enterprise infrastructure to the .NET Framework that solves a good chunk of this problem. What they have done so far is give us the Enterprise Library, which solves some real problems, but is still, as its name suggests, a library. You’ll still need an implementation of an architecture using this library before you can start your application. Why the Enterprise Library isn’t directly part of .NET is another question. I love this quote:

“Enterprise Library is a guidance offering, designed to be reused, customized, and extended. It is not a Microsoft product.”

So, why hasn’t Microsoft solved this yet? I think they don’t want to offend anyone. When one implements an architecture you have to make choices that will cause concern and questions from almost any of the consumers. Where is the application metadata stored – relational database or the file system? Is the object mapping strictly typed or not? What is the external access strategy – Web Services? What about authentication? Permissions? Data Permissions? Deployment? Testing? Localization? Internationalization?

The other reason there isn’t any cohesive strategy around an implementation of an enterprise architecture is because there are a lot of people that get paid a lot of money who are pretty interested in continuing to get paid to design and implement terrible applications. If I had a choice between a year getting paid $200 (or more) an hour or simply using an available solution, it would be a simple choice.

The other problem is that depending on the group of people you are talking to, many are convinced that this problem can be solved by using the right tools or selecting the correct grid control. Really. I’ve spoken to presumably smart people who have told me that the problem with enterprise development is simply management process. This is generally followed by a conversation with someone who knows that if you would simply get the requirements right nothing would go wrong. Next, I’ll speak to an Architect who has landed on a architecture that is, in its entirety, .NET. And, of course, you can throw a rock and hit someone who believes that the answer is UML. The worst, of course, is this: “we’re using SOA.”

I-got-a-good-idea

NetQuarry is an enterprise application platform (for Microsoft .NET). What this means is that we have a layer of runtime code that reads a definition of your application at runtime and produces an ISV quality web-based application. The definition includes references to code that you write (extensions), profiles you declare, schema that you design, and presentation that you build. Much of the definition is built using our tooling (NetQuarry Studio). When you write your code you use Visual Studio, we’re not in the IDE business. Read more here.

The important point to understand about NetQuarry is that it is both an architecture and an implementation of an architecture. It is also important to understand what it is not. It is not a code generator and it is most decidedly not a framework.

We also think that the real promise of NetQuarry isn’t simply an architecture with implementation, but rather what you might do with this implementation once it becomes generally available. The largest idea is the concept of application content. What I mean by this is the availability of functional application parts that can be deployed into a platform (like NetQuarry). Real application assembly. Salesforce.com has done something like this with AppExchange. I like AppExchange but of course I can’t see how I’d use this to create OptionEase, DiForza, or iLUXCARS. For these sorts of applications I have to build it myself. So, the content parts should be a bit less coupled with the application, which is hard, but possible, again, once we have a real implementation of a platform.

If you are interested in the NetQuarry platform or what you might be able to do with it please feel free to contact us: info@netquarry.com.

Ryan