Wednesday, December 23, 2009

Business Process Modeling for Software Developers

As a software developer I never thought I'd be saying this (I suppose eight years of working for the company that invented this technique might bias me), but you can not underestimate the value of business process modeling when starting a new project. This is especially true if it's a small project and you don't have the benefit of a dedicated requirements analyst.

As a consultant (or as a person who is learning someone else's business) your job is to understand your customers existing business better than they do themselves. And you need to be on the same page with others regarding how you are going to change their day to day functions. And that's what business process modeling is for.

You may ignore this document after the first week on the project, in fact I would encourage you to. But even if you throw it away immediately after creating it, the process of developing the document will still:

  • Flush out important questions
  • Show your customer you understand their world
  • Help document the project to other developers
  • Facilitate communication (especially with the people who pay the bills)
  • Generate user stories (requirements)
  • Identify the entities that can feed into an Entity Relationship Diagram (or database I suppose)
  • Aid making good choices for the decisions it be hard to undo later (like whether to manually code workflows or use Windows Workflow Foundation)
  • More clearly identify pain points and areas where your software can help end users; and
  • Identify metrics that can help determine project success from an ROI and product owner's perspective

Convinced this is a tool you need in your toolbelt yet? As long as the answer isn't "I only write code leave me alone" then check out this quick how-to:

Start with the AS-IS diagram

As soon as possible on a new project you should plan to interview your future end users. This shouldn't be an unstructured Q&A session. You should start with a rough draft of what you think their business process is and ask questions that identify the places where you were wrong.

In order to start the diagram you need to identify both the end users and existing systems and put those in "swimlanes" down the left side of a diagram. Swimlanes are important because they help you be explicit about who is performing what activities. Then flush out activities in the happy path in appropriate swimlanes like so:

As soon as you talk to end users (like the clerk in the example above) you will discover unhappy paths. What if the warehouse is out of a product? What if the payment was declined? What happens if a customer needs to return something?

Obviously you document these using standard activity diagram notation:

Identifying Pain Points/Opportunities

The interesting part about unhappy paths is how often they occur, how they affect the customers of your customer, and ultimately how they affect your customer's bottom line. For instance, how often is payment declined in the example above? Fairly frequently? Can you get an actual percentage? If it turns out it's because the original application failed to validate credit cards, then a small amount of programming effort can provide a large benefit to length of time until order fulfillment. Sometimes you can make the process more efficient without writing a line of code.

And at the end of your project if you can show that you've decreased the average length of time it takes to go through the process, ideally by showing less time was spent in unhappy paths, you will likely have made yourself one happy customer. This can be especially effective if your new system can show statistics (like percent of orders that are declined) and compare those to the old process.

Developing the TO-BE diagram

AS-IS process diagrams are completely useless unless you can effectively describe what the process will look like after you implement your new system. The process of developing these documents is pretty similar to the AS-IS diagram except of course you're trying to fix the pain points.

One technique I like for generating a backlog of user stories (tasks) is to put unique numbers with each step in the TO-BE process, then ensure there is at least one user story per step in the process. You may be surprised at how easily activities in your new project's swimlane map to user stories.

Summary

I'm a fan of writing code as soon as possible on a project, but if you fail to understand an end user's business process you may be writing the wrong code. Business process modeling can ensure you're writing the right code to solve the most important problems first. And that may be the difference between project success and project failure.

Wednesday, November 18, 2009

Ruby on Rails, a Microsoft Developer's Perspective

I suppose I'm a pretty hard core Microsoft developer. I've been doing Microsoft .Net C# development since the .Net framework was first released in 2002. Before that, starting in 1997 about when the technology was first released, I developed in Visual Basic with classic ASP (Active Server Pages). So that's twelve years of Microsoft development all told. Which is why the Ruby on Rails class I've been taking this week is so interesting. Ruby represents a significant branching out for me. So these are my initial impressions. I am a complete Ruby neophyte, so please keep all flames at least friendly in tone :).

Dynamic Typing

Let's get to the heart of this Ruby stuff. I thought I would passionately hate dynamic typing. I was getting ready to fill pages ranting about its evils. But that moment never came. Don't get me wrong the Visual Studio strongly typed intellisense no-reading-documentation-needed experience is far nicer than the NetBeans look-it-up-in-irb-with-.methods experience. But the feeling was more like mild to moderate annoyance rather than unbridled hatred. Like when I mistyped a variable and Ruby decided it was a new variable initialized as nil. Annoying, and hard to track down, but hey I'm a Ruby newbie, experts probably don't make these mistakes, right? Anyway, I would be interested to hear how non-trivial, well tested applications perform with refactoring (e.g. variables/fields in views?) and how bad things get during O&M.

Speed of Development

I figured speed of development would be Ruby's saving grace. It was that and more. The Ctrl-S-Alt-Tab-Ctrl-R-zero-delay development cycle is frickin' amazing! It made development feel more fun than it has in a long time. Part of that may be the slower-than-molasses-in-December speed of development I experience today on my (not 2010) SharePoint project. Regardless, even if I were working in an ASP.Net MVC project I suspect the extra speed of development in Rails by skipping the main compilation and JIT compilation steps would increase the enjoyment of a Ruby project enough to equal out the lack of strong typing.

Active Record vs LINQ

Perhaps I'm just not experienced enough with active record, but to me the syntax feels contrived, non-intuitive, and just kind of agitating. I mean it's cool that you can write

p = Person.find_all_by_first_name_and_last_name("Lee", "Richardson")

Ugly as sin, but cool. The alternative doesn't feel much better:

p = Person.all(:conditions => { :first_name => "Lee", :last_name => "Richardson" })

Uch. Of course I am biased. I'm passionate about LINQ. I love LINQ more than any single feature in Microsoft development. I think LINQ may be the most brilliant stroke of genius Microsoft (Anders Hejlsberg) has ever had. And I don't think I realized how passionately I felt about the technology until I didn't have it. So for me I'll stick with:

Person p = ctx.People.Where(p => p.FirstName == "Lee" && p.LastName == "Richardson");

Duck Typing

My mind was blown when the instructor showed us polymorphism without inheritance. I can't argue about how incredibly powerful it is. I just cringe to think of the potential misuse. But without any real Ruby experience I'll have to just leave it at "Wow!"

Mocking

It's pretty scary that you can override the functionality of any method anywhere in Ruby. But when the instructor showed us overriding DateTime.now for the purposes of mocking I had an "ah ha!" moment. Mocking DateTime.Now in C# is an awful experience that involves an intermediate class with a virtual "Now()" method. Ruby sure got C# on that one.

YAML

It's no secret that I passionately hate XML. And it's no secret that Microsoft passionately loves XML. Pity about the mismatch. Ruby on Rails really endeared itself to me when its designers recognized that XML is a travesty against humanity and used YAML Ain't Markup Language instead. Nice!

Interactive Ruby Console

The Interactive Ruby Console (IRB) is just wonderful. Now I can at least get that functionality today in C# with Joe Ferner's excellent Developer's Toolbox. What I can't get is the Interactive Rails Console (ruby script/console). Now that is awesome. I can't wait for C# 4.0 which I suspect will have this. For now RoR++, C#-- .

Tooling

I'm not sure how related this is to dynamic typing, but surprisingly I really, really missed Visual Studio. Perhaps RubyMine is better that NetBeans. I sure wasn't impressed with Komodo. I just felt like constantly switch between my IDE and various console windows, and a database viewer felt clumsy yet somehow necessary with Rails. It just felt so 1999.

Summary

There's no doubt about it, I could be very happy on a Ruby project. The no XML thing is just a perk, the fast development cycles and bringing fun back to coding again is truly awesome. But am I ready to give up Microsoft development full time just yet? No way. LINQ, strong typing, and believe it or not Visual Studio tip the scales back to about equal. Now if only Microsoft could somehow make compilation instantaneous.

Friday, October 23, 2009

SharePoint 2010 Expectations Meet Reality

The current version of SharePoint and I have a love hate relationship. Since I want to be able to look forward to a more joyful and fulfilling relationship with the tool/platform, I made a list of my biggest frustrations before I left for the SharePoint 2010 conference (#spc09). My objective was to return with most of the items checked off as resolved and to have a bunch of new functionality that I hadn’t realized I needed. SharePoint 2010 easily fulfils the second promise with stuff like Business Data Services (Awesome!) and Visio Data Services (Amazing!). But I don’t want to try to recap the whole conference so here’s just how it compares to my initial list:

Testability

SharePoint is notoriously hard to unit test. The good news is Microsoft provided a whole session devoted to software development best practices that included a good chunk on unit testing. It was wonderful to see!

The bad news: SharePoint 2010 will not have public constructors and classes will still be sealed (final). The good news to the bad news: After discussing this issue with Chris Keyser I discovered that Peli de Halleux of Microsoft Research has a free tool called Pex that solves the whole problem. It allows what it calls detours, which allow you to mock non-virtual methods. This stuff is amazing and I’ll blog more about it later.

Summary: Testability is better even for existing SharePoint installs -- so between Sporm and Pex this is a non-issue -- I’m happy.

No Referential Integrity

It kills me that if you create an employee list item that references a company list item and you delete the company list item then nothing happens. No cascading delete. No error message. Just an orphan record. SharePoint 2010 now supports both true referential integrity through errors and cascading deletes!

Summary: Awesome!

Validation is Terrible

Ever tried to validate by pattern in SharePoint (e.g. social security number)? How about range validate (e.g. date of birth can’t be after today)? Or maybe validate one field against another? The answer is you can’t. Not easily or consistently. If you handle validation in an event receiver then the user doesn’t get the error until after a postback on a separate page. And custom field controls are a lot of work and they don't work in datasheet view.

SharePoint 2010 mostly solves these validation issues by allowing field level and list level validation. The error messages even show in datasheet view. The bad news is that pattern matching is not currently supported (e.g. you can’t validate social security number).

Summary: Good stuff, need to see the final version to be extremely happy.

Poor Documentation

Ever noticed that blogs give you better details on SharePoint APIs than MSDN does? I don’t even bother looking on MSDN for SharePoint content anymore. Since the hands on labs didn’t contain any documentation I’ll have to refer to what Steve Ballmer said during the Q&A session, which was that better documentation will be a priority.

Summary: high hopes, but TBD.

Views Don’t Allow Precedence

If you’ve ever needed a view with A and (B or C) and ended up with (A and B) or C, you know where I’m coming from. As a developer you can implement precedence in CAML, but as an end user you're stuck. Sadly this issue is still not fixed in SharePoint 2010.

Summary: Fail.

Re-Deployment is Hard

Currently redeploying lists is hard; redeploying web parts without overwriting user settings is hard; and redeploying workflows is very hard.

Without documentation and intermittent Internet Joe Ferner and I had a hard time manually confirming that this is better. We also missed a key session on Thursday that talked about it. There were some tweets that looked promising, but I’ll have to wait until Thursday’s presentations are posted to be able to comment for sure.

Summary: Good I think, but still TBD.

Massive Duplication in CAML Instantiated Lists

Using CAML to instantiate a list is currently awful in SharePoint. It’s like 5 pages of CDATA and HTML and XML with massive duplication. Sadly SharePoint 2010 doesn’t solve the root problem: list instantiation looks exactly the same. The good news is that along with just about every other aspect of SharePoint 2010 development, Visual Studio makes this task mind numbingly easy to do. If you can just try not to look in that file that was auto-generated for you, then you should be fine.

Summary: Better.

Poor Usability

Once you get used to the current version of SharePoint the user interface is extremely consistent and many users really like it. But if you were to compare it to any modern Web 2.0 site it's a complete failure. SharePoint 2010 does an excellent job of implementing selective refresh/AJAX/Web 2.0. But to me it feels more complicated. Maybe this is because it introduces the ribbon, which I have never liked. Maybe it just is more complicated. In any event I'm not the best judge of usability, so I’ll have to wait and see what others say.

Summary: Definitely better, but jury’s still out.

Can’t Reference a List AND Content Type

Suppose you have a Calendar list. The list supports two content types: Events, and Iterations (sprints). It’s a nice architecture because you want to view iterations and events in the same calendar views. Now if you have a User Story list, wouldn’t it be nice to have a lookup field that points only to Iterations? Sadly you couldn’t do this before and you won’t be able to for the foreseeable future.

Summary: Fail.

CAML for Queries

Technically this wasn’t on my list because I use Sporm, but on behalf of my non-spormified software development brethren let me say that writing queries in CAML is awful.

SharePoint 2010 makes enormous strides in this area. I won’t go into great detail, but you can now use LINQ on the server side and on the client side!

Summary: Awesome!

Summary

Lots of TBD, lots of awesomeness, still some fail. Worth upgrading? Absolutely. Will it still be a love hate relationship? Probably, but at the moment it’s looking pretty darn good.

Sunday, September 13, 2009

Sporm - Out of the Depths of SharePoint's XML Hell

In my last post I described how a new open source tool called sporm significantly simplifies unit testing SharePoint. Making SharePoint unit testable is my absolute favorite feature of sporm because SharePoint is notoriously hard to unit test. But sporm provides other benefits as well and its ability to pull us out of the depths of verbose loosely typed XML hell and into LINQ excellence is next on my list of favorite features. So in this post I’ll describe the pre-sporm technique of querying with CAML, how to query data using sporm, and finally how sporm supports SharePoint’s unique architecture of allowing multiple content types per list and what that means to you.

Caml’s Are Ugly

Warning: if you’re new to SharePoint then what you’re about to see may shock and upset you. If, like me, you hate both XML and loose typing then you will agree that CAML is awful, but bear with me I promise sporm will make it better. Much better.

CAML or Collaborative Application Markup Language is how one queries for data in SharePoint. A simple query might look like this:

<Query>
  <
Where>
    <
And>
      <
And>
        <
Eq>
          <
FieldRef Name='First_Name' />
          <
Value Type='Text'>Lee</Value>
        </
Eq>
        <
BeginsWith>
          <
FieldRef Name='Last_Name' />
          <
Value Type='Text'>Rich</Value>
        </
BeginsWith>
      </
And>
      <
Leq>
        <
FieldRef Name='Dob' />
        <
Value Type='DateTime'>2009-01-01T00:00:00Z</Value>
      </
Leq>
    </
And>
  </
Where>
</
Query>

Simple right? ;) In case you didn’t catch the meaning from the slightly, uh verbose, query this asks for records with a First_Name of “Lee”, a Last_Name starting with “Rich” and a “Dob” less than or equal January 1 2009.

There are a couple of things to note about this query:

  • Field names are loosly typed. If Dob were ever renamed to DateOfBirth the query would fail at runtime (probably during a demo to a customer, or if you’re lucky during integration tests), but certainly not at compile time.
  • And is a binary operator. This forces explicit precedence and removes the need for parenthesis, but at the cost of readability.
  • Types must be explicitly defined. I guess this is necessary since it’s XML, but somehow I just don’t feel like this should be necessary.
  • It’s XML. Ok obviously, but the point is you have to type everything twice. <BeginsWith> </BeginsWith>. Ouch, so verbose, so angley, I so hate XML.

Now, there are tools that make this better. U2U’s free CAML Query Builder tool significantly improves the experience of querying SharePoint data.

But it makes you wonder if something is wrong when you need a tool to retrieve data from your data store. Do you typically use tools to assist when you’re writing SQL? Probably not. But like I said hang in there, sporm make things better.

Unlinq my Caml

If you were to write the same query as above using sporm it would look like this:

IQueryable<Employee> employees = GetEmployees().Where(e =>
      e.FirstName == "Lee"
      && e.LastName.StartsWith("Rich")
      && e.Dob <= new DateTime(2009, 1, 1));

Just a little easier to read than CAML, right? A couple of things to note:

  • Fields are strongly typed. If you were to rename FirstName the compiler would catch every single instance at design time.
  • Operators are standard C# operators. &&, .StartsWith(), and <= are all familiar and concise and use a standard, known precedence.
  • Types are standard C# types. You never have to explicitly say that "Rich" is a string, it just is.
  • Your query is actual C# code. The lambda (closure) and the fact that it uses deferred execution might throw off a junior developer in some scenarios, but the query is readable and works exactly the same as if you were querying in memory objects or querying a database with LINQ to SQL or the Entity Framework.

Nice! What's beautiful about this is that sporm converts your C# directly into CAML using C# 3.0's expression trees feature. What sporm can't convert to CAML it executes in memory transparently to you. Sporm uses log4net and outputs its CAML queries to the console by default, so it is a good idea to watch the output if you're concerned about performance.

Now the following isn’t relevant to the comparison with CAML, but I would be negligent if I didn’t explain how the GetEmployees() function works.

SP != DB B/C of Content Types

First of all GetEmployees looks like this:

private IQueryable<Employee> GetEmployees() {
      return MySPDataContext
            .GetCurrent()
            .GetList<Employees>()
            .OfType<Employee>();
}

How it works is that the static GetCurrent() method retrieves sporm’s context object, which knows how to query a SharePoint site, from either the web context or thread local storage; next the GetList() method tells sporm which list you want to query; and the OfType() method tells sporm which content type within the list you want to query. This last part is important because sporm supports SharePoint’s ability to have multiple content types per list, which other LINQ providers like LINQ to SharePoint do not. But what are content types and why should you care?

SharePoint’s List/Content Type architecture seems odd at first, but it allows interesting scenarios not available in a traditional database. For instance you might have a calendar list that contains multiple types of records (list items) in the same list. Your calendar list might contain some combination of the following three record types: meetings with unique fields like Organizer (a person); iterations with unique fields like DeployedToProduction (a Boolean); and actions with unique fields like RelatedEmployee (a reference to another list). This architecture allows SharePoint to view all three types of records in a single view: like a per month calendar view. The data might look like this:

Title Start End ContentType Organizer Deployed To Production Related Employee
Iteration16 1/12/09 1/19/09 Iteration   false  
Stakeholder Demo 1/19/09 3 PM   Meeting Lee Richardson    
Tag Trunk 1/19/09   Action     Lee Richardson

Sporm’s unique architecture supports this scenario by allowing you to retrieve iterations like this:

return MySPDataContext
      .GetCurrent()
      .GetList<Calendar>()
      .OfType<Iteration>();

Or get meetings from the same list like this:

return MySPDataContext
      .GetCurrent()
      .GetList<Calendar>()
      .OfType<Meeting>();

While you may only use multiple content types per list occasionally you can feel comfortable knowing that sporm will support you when you need it. The rest of the time you can arrange your architecture to accommodate your 90% scenario of one content type per list. I’ll discuss the architecture I’m using on my current project in my next post.

For now I hope this has clarified some of the benefits of using sporm, and I hope that you’ll consider using it on your next SharePoint project.

Wednesday, September 9, 2009

Unit Testing SharePoint - Past, Present, and Sporm

As I described in SharePoint: The Wild West of Software Development there is a serious problem when you develop for SharePoint: ensuring quality through unit testing is really, really hard. And that's where a new open source tool just released today called sporm (SharePoint Object Relational Mapper) comes in. While sporm provides many benefits besides simplified unit testing I wanted to focus on this topic first, because sporm's approach, which models the entity framework in the way it supports POCO's, is a unique feature not available with other SharePoint tools like LINQ to SharePoint.

I'll start by describing the unit testing problem, then provide a conventional solution with mocking, then finish with the elegance of a sporm based solution.

Simple Unit Testing Turns Ugly

I would bet anyone that attempts to develop a tiered solution with SharePoint ends up with entities that look something like this:

public class Employee {
      public SPListItem ListItem { get; private set; }

      /// <summary>
      /// The constructor requires that you pass in a list item
      /// that is used by your strongly typed properties
      /// </summary>
      public Employee(SPListItem listItem) {
            ListItem = listItem;
      }

      /// <summary>
      /// This provides your entities strong typing to hide
      /// the native weak typing in SharePoint
      /// </summary>
      public virtual string LastName {
            get { return (string)ListItem["LastName"]; }
            set { ListItem["LastName"] = value; }
      }
}

And if you have some function that you'd like to unit test like this:

/// <summary>
///
Typically returns "[LastName], [FirstName]" but skips
/// the comma if either is missing
/// </summary>
public string GetNameFormatted() {
      bool noFirst = string.IsNullOrEmpty(FirstName);
      bool noLast = string.IsNullOrEmpty(LastName);
      if (noFirst && noLast) return "";
      if (noFirst) return LastName;
      if (noLast) return FirstName;
      return string.Format("{0}, {1}", LastName, FirstName);
}


Now you face a major problem: LastName is tightly coupled with SPListItem. And SPListItem only contains internal constructors making it unmockable. If you keep the tight coupling and pass an SPListItem into the constructor then you need a connection to SharePoint and an employee list (table) and you need to create an employee list item (record), then you test it, and finally you ought to clean up after yourself and delete the list item. Talk about a simple unit test turned fragile integration test. Yuck.

Mocking the Solution

So at this point you're undoubtedly thinking it's time to mock. To do this you'll have to make a public constructor that takes no arguments and mark your properties virtual. Then you can write a test like this:

[TestMethod]
public void TestGetNameFormatted_FirstAndLastExist_LastCommaFirst() {
      MockRepository mocks = new MockRepository();
      Employee mockEmployee = mocks.StrictMock<Employee>();
      Expect.Call(mockEmployee.FirstName).Return("Lee").Repeat.Any();
      Expect.Call(mockEmployee.LastName).Return("Richardson").Repeat.Any();
      mocks.ReplayAll();
      string actual = mockEmployee.GetNameFormatted();
      mocks.VerifyAll();
      Assert.AreEqual("Richardson, Lee", actual);
}

And then if you're like me you absolutely hate your unit tests. Because let alone that you had to modify your production code to accommodate unit testing, more importantly your unit tests are now unreadable. All of them. Even in simple methods without dependencies your unit tests contain almost as much plumbing code as actual test code. Enter sporm.

Unmocking with Sporm

Sporm is an open source tool released on codeplex that was developed by fellow Near Infinity employee Joe Ferner (who, incidentally, is one of the most brilliant developers I know). While this is a version 1.0 product Joe has been using it on his project and independently I've been using it on my project for several months now, so it is relatively mature.

So how it works is that at design time sporm will read from your SharePoint site and generate partial classes for each of your list items and each of your content types. The generated classes don't inherit from anything, giving you the ability architect your solution how you see fit. And the properties are auto-properties, making the classes just pure POCO. Here's an example if you're interested:

[SPOrmContentType(Name="Employee")]
public partial class Employee : ISPOrmContentType {
      public static class FieldTitles {
            public const string FirstName = "First Name";
      }

      public static class FieldStaticName {
            public const string FirstName = "FirstName";
      }

      [SPOrmIdField()]
      public virtual int Id { get; set; }

      [SPOrmField(Title = FieldTitles.FirstName, StaticName = FieldStaticName.FirstName)]
      public virtual string FirstName { get; set; }
}

Since generated entities look like that, your entities can now look like this:

public partial class Employee {
      public string GetNameFormatted() {
            bool noFirst = string.IsNullOrEmpty(FirstName);
            bool noLast = string.IsNullOrEmpty(LastName);
            ...
      }
}

Now if that isn't a picture of beauty to you then you haven't been working with SharePoint. But wait, I hear you asking, how does this have anything to do with SharePoint? How would an instance of an employee be able to retrieve the data in a SharePoint ListItem? The answer is in the DataContext. In your production code you write something like this:

public static Employee FindById(int id) {
      return MySPDataContext
            .GetCurrent()
            .GetList<TList>()
            .OfType<TContentType>()
            .FirstOrDefault(c => c.Id == id);
}

While it might not look so pretty at first you can hide these details in a base type and the nice thing is that it does handle SharePoint's ability to have multiple content types per list (which LINQ to SharePoint does not do, incidentally). But for now all you need to know is that when you return an Employee thorugh MySPDataContext it subtypes your Employee class and returns you a proxy at runtime. The proxy overrides each property to return you the appropriate value in the SPListItem. And if you instantiate an Employee directly (as you would in unit testing) then you can use it like a pure POCO. If you're familiar with the entity framework this should sound extremely similar. The result is that you can write your unit tests like this:

[TestMethod]
public void TestGetNameFormatted_FirstAndLastExist_CommaSeparate() {
      Employee employee = new Employee {
            FirstName = "Lee",
            LastName = "Richardson"
      };
      string actual = employee.GetNameFormatted();
      Assert.AreEqual("Richardson, Lee", actual);
}

No SharePoint dependencies. No mocking. Nothing but pure, unadulterated unit test. Now that a thing of beauty.

Monday, August 17, 2009

SharePoint: The Wild West of Software Development

Some argue that Microsoft developers lack rigor -- that techniques like unit testing and continuous integration are virtually unheard of in this space. That’s rubbish. I would be shocked if the percentage of developers that track code coverage is significantly different for the .Net or Java spaces (not quite mainstream, but respectable). Mind you I’m talking .Net development in general. SharePoint, now that’s another story.

The shadowy world of SharePoint is one in which source control is a distant memory, where no distinction is drawn between development, test and production environments, and where roaming bandits take over small towns with no fear of the law.

Why is this? How has it come to be? The answer is that for the most part SharePoint treats code and data as one in the same. End users can modify data stores: they can add, delete and modify tables (Lists) and columns (Fields) at will. They can open up SharePoint Designer and modify pages, customize workflows, rob banks, and run local Sherriff’s out of town.

And that’s why end users love SharePoint. It removes the slow, burdensome bureaucracy involved in doling out development resources from large centrally planned corporate or government IT departments and places that power directly in the hands of end users, allowing them to get their jobs done faster and to adapt to change quicker. And that is why I maintain that as software developers SharePoint, or a product like it, is in our future -- like it or not.

But the power given to end users makes rigor and good design extremely hard for developers because the production machine you deployed to last month may look completely different now that you’re ready to redeploy. And Microsoft makes good design even harder with an inflexible API. Critical classes like SPSite (think SqlConnection) contain no public constructor, rendering them completely unmockable (unless you’re willing to spend $450 per developer for some TypeMock Isolator magic). And vital classes like SPList (think DataSet) are marked final, crippling your ability to make nice strongly typed entities in your architectures.

So what is a Type-A, quality minded software developer to do?

The thing to keep in mind is that the challenges that SharePoint places on developers and architects simultaneously make good coding practices harder (but not impossible) while making them more important than ever.

Because that column you thoughtfully added as required may get reset to optional by an end user in production, you need to write more good tests that validate once solid assumptions. Because you can’t inherit SPList, you’ll need to encapsulate it, requiring more code per entity. Because you can’t mock essential SharePoint classes you will need to write more integration tests instead of unit tests. Because you’re writing integration tests your tests will be more fragile and you’ll spend more time fixing the test code than the real code.

And because you’re spending more time focusing on quality while your Wild West, short term focused competition in the SharePoint world slings code together, they will appear faster and better able to get applications out the door. It can be frustrating, because over time your well designed and well tested application will be more maintainable and will cost IT departments less. But end users frequently don’t understand this, so as long term minded SharePoint developers we must focus on educating end users.

Combat the short term focused mindset by keeping everything in .wsp files, storing code in source control, avoiding the temptation to apply fixes in production, using continuous integration (yes, for SharePoint!), and most importantly tracking code coverage (yes, code coverage for SharePoint code!). Put quality metrics in status reports for management to see. Track defects and make your stats publicly available (since you’ll have significantly fewer over time than code slingers, and you want end users asking for stats). In short remind the world that quality counts, even in the shadowy realms of SharePoint. If we work together we can bring law and order to the … um, West.

Tuesday, August 4, 2009

You cannot grant a user a permission level that is not attached to a web

If you receive this error message that looks like this:

Microsoft.SharePoint.SPException: You cannot grant a user a permission level that is not attached to a web.
  at Microsoft.SharePoint.SPRoleDefinitionBindingCollection.AddInternal(SPRoleDefinition roleDefinition)
  at Microsoft.SharePoint.SPRoleDefinitionBindingCollection.Add(SPRoleDefinition roleDefinition)

When you’re doing something like this:

private static SPRoleDefinition FindAddEditListItemsRoleDefinition(SPWeb site) {
    SPRoleDefinition definition = site.RoleDefinitions
        .Cast<SPRoleDefinition>()
        .FirstOrDefault(def => def.Name == ROLE_DEFINITION_TITLE);

    if (definition != null) return definition;

    definition = new SPRoleDefinition {
        Name = ROLE_DEFINITION_TITLE,
        Description = "Can view and edit list items only.",
        BasePermissions = SPBasePermissions.EditListItems
            | SPBasePermissions.ViewListItems
            | SPBasePermissions.ViewFormPages
            | SPBasePermissions.ViewPages

            | SPBasePermissions.Open
        };
    site.RoleDefinitions.BreakInheritance(true, true);
    site.RoleDefinitions.Add(definition);
    site.Update();
    return definition;
}

It may be because SharePoint doesn’t allow you to use the SPRoleDefinition you just created and you have to return the one you just added. Stupid, but hopefully it will help someone somewhere.

    ...
    site.Update();
    // you can't just return definition because SharePoint gives you
    //    "You cannot grant a user a permission level that is not
    //    attached to a web" when you attempt to use it, so
    //    you need to return it again
   
return site.RoleDefinitions
        .Cast<SPRoleDefinition>()
        .First(def => def.Name == ROLE_DEFINITION_TITLE);
}

Thursday, July 23, 2009

IEnumerable.Count() is a Code Smell

Way too often I come across code like this:

IEnumerable<Product> products = GetProducts();
if (products.Count() == 0) {
      SomePlaceHolder.Visible = false;
} else {
      SomePlaceHolder.Visible = true;
      SomeRepeater.DataSource = products;
      SomeRepeater.DataBind();
}

What’s wrong with this is that Count() is a LINQ extension method that literally iterates through every item in an enumerable. In other words it won’t short circuit after it finds more than zero items -- it will continue iterating through every single item. And while that may be a fast operation in dev, it may be a really slow operation in production.

But it gets worse. Because of LINQ’s deferred execution the results of .Count() are not cached. So when there is at least one product you are guaranteed to enumerate every single product twice: once for .Count() and once for .DataSource = products.

Do Any() Solutions Exist?

One possibility is to instead use IEnumerable.Any(). So you could rewrite the above code like this:

IEnumerable<Product> products = GetProducts();
bool anyProducts = products.Any();
SomePlaceHolder.Visible = anyProducts;
if (anyProducts) {
      SomeRepeater.DataSource = products;
      SomeRepeater.DataBind;
}

The results will be better regardless of if your data source is in memory or a database, but for illustration purposes LINQ to SQL outputs the following SQL if you run the code smell version when there is at least one product in the database:

SELECT COUNT(*) AS [value]
FROM [dbo].[Product] AS [t0]
SELECT [t0].[ProductId], [t0].[ProductName], ...
FROM [dbo].[Product] AS [t0]

And it outputs this SQL if you run the better version when there is at least one product:

SELECT
    (CASE
        WHEN EXISTS(
            SELECT NULL AS [EMPTY]
            FROM [dbo].[Product] AS [t0]
            ) THEN 1
        ELSE 0
     END) AS [value]
SELECT [t0].[ProductId], [t0].[ProductName]
FROM [dbo].[Product] AS [t0]

Looks a little uglier, but if you run the execution plan you’ve converted two table scans into an index scan and a table scan (well, depending on your schema of course). Regardless, much better execution-wise.

Undeferring deferred execution

Now what if your data source is a database and you really want to optimize the heck out of this and only perform one SQL statement? You could disable deferred execution by doing this:

IList<Product> products = GetProducts().ToList();
bool anyProducts = products.Any();
SomePlaceHolder.Visible = anyProducts;
if (anyProducts) {
      SomeRepeater.DataSource = products;
      SomeRepeater.DataBind;
}

The .ToList() function forces enumeration to happen immediately and cache the results in a list. This achieves the result of converting two queries into one, but be warned it also destroys one of the nicest benefits of LINQ’s deferred execution: the ability to append .Sort() or .Skip().Take() (paging) statements at the very end and have them execute in your original query, in SQL, in the database, where they belong.

Summary

IEnumerable.Count() seems like a pleasant and friendly convenience function; it feels so similar to the innocuous List.Count -- but don’t be fooled. It’s a mistake. One you (or the person who maintains the system when you leave) may soon regret.

Sunday, May 17, 2009

Secrets of SharePoint 2010 Exposed at TechEd 2009

Update 3/22/2010

This seems to be a popular post for those looking for information about AJAX applications in SharePoint 2010. If this topic interests you definitely check out my multi-part series Client Side AJAX applications in SharePoint 2010.

In case you aren't aware, the details of SharePoint 2010 are under tight wraps. If you asked any of the presenters at Tech Ed the typical response was "No idea what you're talking about." But that doesn't mean presenters didn't occasionally slip up or say more than they probably should have.

Publically Stated SharePoint 2010 Info

Here’s what has been publically stated about SharePoint 2010 thus far:

  • Requires at least Windows Server 2008 64 bit
  • Requires SQL Server 64 bit (2005 or 2008)
  • Will not support for IE 6
  • Public testing to occur first half of next year

This was announced at TechEd by Tom Rizzo, director of Microsoft SharePoint Server, as posted in this InfoWorld article and also on the SharePoint blog.

New SharePoint 2010 Secrets

An exciting slip up that has received no press thus far was from the presentation: Microsoft Visual Studio 2010 Overview for the Business Application Developer. If you check 1:00:33 of the presentation you’ll see the first glimpse of a SharePoint 2010 Site Home Page:

I could tell something wasn’t right when I saw this part of the demo, but fellow Near Infinity employee Joe Ferner put it together first when he saw the big red "Give Feedback" button in the top right. He tweeted about what came next:

"Sharepoint 2010 to have inline list ajaxy editing."

It looks really nice, and is very exciting since SharePoint is currently a little unwieldy. Check out this screenshot from 1:01:08:

So that was the most exciting slip up. Other exciting tidbits were:

SharePoint designer will support saving workflows to re-use for provisioning

I learned this one during the Q&A session of the "Automate Business Processes Using InfoPath Forms with Integrated SharePoint Designer Workflows... All without Coding!" talk. It’s currently a pain point that workflows developed in SharePoint designer can’t be reused in the same way workflows developed in Visual Studio can, so this will be a wonderful new feature.

BDC will probably support updating and inserting data

Apparently MOSS only shipped without this feature because it hadn’t been thoroughly tested. That there has been plenty of time to test this feature by now, so it’s a pretty reasonable assumption that it will be in the next version.

Finally this is pure speculation, but I’m guessing:

SharePoint 2010 to support ASP.Net Data Services API

Microsoft supports two AJAX approaches: client side and server side. The server side uses the UpdatePanel control where you code as usual and everything magically gets AJAXized. The client side version is part of the unreleased ASP.Net 4.0, but ties in tightly with JSON provided by the .Net Data Services API that was released in the .Net Framework 3.5 SP1.

The SharePoint team could have used the server side approach, but why not use the latest technology available, speed up UI significantly, and simultaneously allow web part developers a fantastic level of client side AJAX support? If they made the entire SharePoint API available via the .Net Data Services API it would enable some powerful fast AJAX web parts. It would be awesome.

Finally, the only other tidbit isn’t a secret at all, but what I consider to the most exciting feature of Visual Studio 2010: it will support editing SharePoint solutions without editing XML. I hate XML, so that will be wonderful. So are you ready for SharePoint 2010 today? I sure am. Pity about the wait.

Thursday, May 7, 2009

Code Access Security Cheat Sheet

I did short presentation on Code Access Security (CAS) a while ago and put together a cheat sheet to help remember the terms and how they fit together. It includes screenshots of the .NET Framework 2.0 Configuration tool (in Control Panel\Administrative Tools) and uses Entity Relationship Diagram (ERD) Notation. I had to pull it up recently and figured others might also find it helplful.

It describes the following terms:

  • Permission
  • PermissionSet
  • Code Group
  • Policy Level
  • Assembly Instance
  • Evidence; and
  • Evidence Type

It should print to 8.5 x 11. Click to enlarge:

Hope you find it useful.

Sunday, April 26, 2009

Making SharePoint Title a Calculated Column

All SharePoint lists start with a user editable "Title" column that ECB menu's hang off of and that is the default field to display in associated child tables. This works well most of the time. For example if you have a list like Company, you can change the display name of Title to "Company Name," and then Employee records display that field for their parent relationship.

Where this doesn't work well is lists like Employee, where there is no single column that uniquely identifies a row to an end user. You could create a new "Full Name" calculated column and delete the Title column, but making Title a calculated column just feels like a cleaner solution.

I looked into making Title calculated where it is defined in CAML, but it looked pretty messy, and a true calculated column additionally suffers from the limitation that you can’t pull from other lists. The solution I settled on was to populate it in ItemAdding and ItemUpdating. Of course you tie into those methods with something like:

SPList employeeList = site.Lists["Employee List"];
employeeList.EventReceivers.Add(
    SPEventReceiverType.ItemUpdating,
    "[four part assembly name]",
    "Namespace.EmployeeEventReceiver"
   
);
employeeList.EventReceivers.Add(
    SPEventReceiverType.ItemAdding,
    "[four part assembly name]",
    "Namespace.EmployeeEventReceiver"
   
);

The only reason this is worth documenting is because setting title is a little messy in the event receivers and there is very little documentation on how to do it.

public class EmployeeEventReceiver : SPItemEventReceiver {

  public override void ItemAdding(SPItemEventProperties properties) {
    string fullName = GetFullName(properties);
    properties.AfterProperties["Title"] = fullName;
    base.ItemAdding(properties);
  }

  private static string GetFullName(SPItemEventProperties properties) {
    string firstName = properties.AfterProperties["First_Name"].ToString();
    string middleName = properties.AfterProperties["Middle_Name"].ToString();
    string lastName = properties.AfterProperties["Last_Name"].ToString();
    return Employee.GetFullName(firstName, middleName, lastName);
  }

  public override void ItemUpdating(SPItemEventProperties properties) {
    string fullName = GetFullName(properties);
    string internalTitleName = properties.ListItem
        .Fields["Title"].InternalName;
    properties.AfterProperties[internalTitleName] = fullName;
    base.ItemUpdating(properties);
  }
}

The tricky part is getting the title field by internal name in ItemUpdating vs. getting it normally in ItemAdding. It’s pretty easy once you work through it, but it took me longer than it should have. So hopefully this helps someone somewhere.