Thursday, July 22, 2010

Death to the DAO and How to Test LINQ

Occasionally I hear complaints that LINQ is hard to unit test. These complaints aren’t about LINQ to objects, mind you, they’re specific to the complexities of the flavors of LINQ that turn C# code into something else like SQL or CAML using expression trees. The most common technologies are LINQ to SQL, the Entity Framework, or in my case at the moment LINQ to SharePoint. In this post I’m going to propose a technique that makes testing LINQ not just easy, but downright elegant – assuming you’re ok with extension methods – lots of extension methods. And assuming you’re ready to kill your Data Access Objects (DAO) tier.

The Unit Testing Problem

Any architecture needs a place to put code that finds entities. For instance FindBySocialSecurityNumber(). In a traditional architecture we might put a method like this is in a DAO layer. If so our method will look something like this:

public class EmployeesDao {
    public Employee FindBySSN(Context ctx, string ssn) {
        return ctx.Employees.SingleOrDefault(e => e.Ssn == ssn);
    }
}

So how would we go about unit testing this?

One fairly typical solution would be to use an in-memory database. That approach works if our data store is a database, but it certainly doesn’t work if the data store is something less traditional like SharePoint. But even if our store is a database, we’ll still have the hassle of setting up the in-memory database.

Another solution might be to use a mock Context that returns an IQueryable. But wouldn’t it be wonderful if we could avoid mocking all together?

Killing the DAO

The first question is why we even have a DAO tier to begin with. The original idea was that we wanted a place to put code specific to a particular data store. In other words we wanted to isolate the code that will need to be changed should the data store switch from SQL Server to Oracle. But isn’t that exactly what LINQ does? I’d be pretty surprised if there wasn’t a decent LINQ provider for just about any data store at this point that required more than minimal code changes. So why not embrace LINQ and reconsider alternatives to a DAO tier?

One alternative that I’ve been using for over a month now is to switch to extension methods. To give credit where it’s due the idea originated with a conversation with fellow Near Infinity employee Joe Ferner. And I'm sure the idea isn't particularly original (please post in the comments if you know others that use this approach).

Using this technique our code changes from something like this:

var employeeDao = new EmployeesDao(); // or use IOC of course
employeeDao.FindBySSN(ctx, "111-11-1111");

To something like this:

ctx.Employees.FindBySSN("111-11-1111");

Among other things I find this far more aesthetically pleasing because each of the three elements to the statement represent a subsequent filtering of data. It's a more functional way of looking at things.

We could implement this off of the Employees property of the context if we have control over that (which I don't with spmetal). But if we implement this as an extension method like this:

public static class EmployeeExtensions {
    public static Employee FindBySSN(this IQueryable<Employee> employees, string ssn) {
        return employees.SingleOrDefault(e => e.Ssn == ssn);
    }
}

We now have something that’s considerable easier to unit test.

Testing It

Once we’ve refactored our function as an extension method that filters down the corpus of entities, we can test the code using in-memory objects with a call to .AsQueryable(). For instance:

public void FindBySSN_OneSsnExists_EmployeeReturned() {
       var employees = new [] { new Employee { Ssn = "111-11-1111" } };
       var actual = employees.AsQueryable().FindBySSN("111-11-1111");
       Assert.IsNotNull(actual);
}

Notice we didn’t have to mock anything.

Testability, but at What Cost?

This technique works great for the example above, but how does it scale to harder problems and what other downsides are there?

As far as scalability I’ve found this technique works great for every scenario I’ve run across in the month I’ve been doing it. It works for joins, aggregations, and even for inserts, update, and deletes.

As far as downsides the astute reader may be wondering about mockability. For instance what if we want to mock the call to FindBySSN and give it the exact Employee that will be returned. This scenario is admittedly harder. But what I've found is that far more often than not I don’t really need to mock the types of things that used to live in the DAO tier. Instead I just mock the Employee object off of context to return in-memory objects and make my tests slightly larger in scope. Most of the time I find the larger scope increases the usefulness of the test. In the occasional case where I do really want to mock the "DAO" tier I use a technique described in either this post or this post by Daniel Cazzulino.

Conclusion

Obviously there is more to this architecture, for instance how do you handle insert and update operations? The short answer is it’s easy, but I’ll save that topic for a future post. For now why not give this approach a try? You weren’t really happy with that useless old DAO tier anyway, were you? I say we eradicate it and never look back.

Saturday, April 10, 2010

ASP.AJAX 4 Templates Part 4 - jQuery, and Manual JSON Object Manipulation

Note: This is part 4 of a multi-part series exploring building client side AJAX applications with ASP.Net AJAX 4 Templating and the WCF Data Services (aka ADO.Net Data Services, aka oData, aka Astoria) in SharePoint 2010.

Part 1 – Exploring WCF Data Services in SharePoint 2010
Part 2 – Creating a read-only page with ASP.Net AJAX 4.0 Templates
Part 3 – Writing data back to SharePoint with ASP.Net AJAX 4.0 Templates
Part 4 - jQuery and Manual JSON Object Manipulation

In the previous posts in this series I’ve shown how you can access SharePoint data using WCF Data Services, how you can display that data using ASP.Net AJAX 4.0 Templates, and how you can write data back to SharePoint.

In this post I’ll take writing data back to SharePoint a step further by showing how you can modify non-visible fields by modifying JSON objects directly. More specifically I’ll enable dragging user story index cards on a virtual bulletin board using a jQuery plugin, save the X and Y coordinates back to JSON objects, and then save the updated records back to SharePoint.

Cards on a Corkboard

The first thing I’ll do is style the cards and display them with absolute position. While I’m at it I’ll also reference the draggable jQuery UI plugin. If you’re following along at home just built a custom .js download from the jQuery UI site, drop it in your layouts directory and add a reference. So my PageHead content control now looks like this:

<asp:content
    id="PageHead"
    contentplaceholderid="PlaceHolderAdditionalPageHead"
    runat="server">

    <script type="text/javascript"
        src="../ajax/MicrosoftAjax.js"></script>
    <script type="text/javascript"
        src="../ajax/MicrosoftAjaxDataContext.js"></script>
    <script type="text/javascript"
        src="../ajax/MicrosoftAjaxTemplates.js"></script>
    <script type="text/javascript"
        src="../ajax/MicrosoftAjaxAdoNet.js"></script>
    <script type="text/javascript"
        src="../jquery-ui-1.8.custom/js/jquery-1.4.2.min.js"></script>
    <script type="text/javascript"
        src="../jquery-ui-1.8.custom/js/jquery-ui-1.8.custom.min.js">
    </script>
    <style type="text/css">
        .sys-template
        {
            display: none;
        }
        .userstorycard
        {
            border: 1px solid #777777;
            width: 200px;
            position: absolute;
            cursor: move;
        }
        .carddescription
        {
            font-size: 13px;
            background-image: url('card_bg.jpg');
            padding: 0px 5px 5px 5px;
        }
        .cardtitle
        {
            font-size: 15px;
            font-weight: bold;
            padding: 0px 0px 0px 0px;
            border-bottom: 1px solid red;
            background-color: White;
            padding: 0px 5px 0px 5px;
        }

        ...

The next thing to do is add X and Y fields to the user story list in SharePoint. And finally display the cards with absolute positioning:

<div xmlns:sys="javascript:Sys" class="background">
  <div id="userStories" class="sys-template">
    <div class="userStoryCard"
      sys:style="{{ 'left: ' + X + 'px; top: ' + Y + 'px;'}}">

      <div class="userStoryTitle">{{ Title }}</div>
      <div class="userStoryBody"><div>{{ Description }}</div>
    </div>
  </div>
</
div>

Notice the sys:style attribute. The sys: part is the namespace you need to add for attributes when using templating that I mentioned in part three. Also notice the JavaScript string concatenation inside the binding. You can put in any JavaScript in there that you like. So now if you manually set X and Y values in SharePoint for a card the card shows in the right place. But we’re still missing the ability to drag cards.

Making Cards Draggable with jQuery

In order to enable dragging support you’re supposed to call the draggable() function on the DOM elements you want to make draggable. Simple right? You just call $(".userstorycard").draggable() which should find every element with a class of userstorycard and call draggable() on it.

But if you do this onLoad() then ASP.Net AJAX Templating engine hasn’t had a chance to render the new DOM elements yet. Fortunately the DataView has the JavaScript equivalent of an OnRendered event that you can tie into:

function onLoad() {
    dataContext = $create(
        Sys.Data.AdoNetDataContext,
        { serviceUri: "/demoprep/_vti_bin/ListData.svc" }
        );

    dataView = $create(
        Sys.UI.DataView,
        {
            autoFetch: true,
            dataProvider: dataContext,
            fetchOperation: "UserStories",
            fetchParameters: { $top: 20 }
        },
        {
            rendered: onRendered
        },
        null,
        $get('userStories')
    );
}

function onRendered() {
    // from http://jqueryui.com/demos/draggable/
    $(".userstorycard").draggable();
}

And that’s it for enabling dragging. Pretty simple. The result looks like this:

Ok, well you can’t see the dragging support, but it’s there, trust me. And it’s pretty cool except for one thing.

Saving Non-Visible Properties

The problem is that every time you refresh the page the nice layout you’ve done disappears and everything goes back to being stacked one on top of the other in the upper left. My first thought was to create input type=hidden form elements on the page. This approach causes a lot more plumbing code than is necessary. The better way is that you can access the underlying in-memory JSON objects. Step one is to register an “event” for when the user stops dragging a card.

function onRendered() {
    // from http://jqueryui.com/demos/draggable/
    $(".userstorycard").draggable({
        stop: onDragStop
    });
}

Once you wire that up the onDragStop() function is where all the interesting stuff happens:

function onDragStop(event, ui) {
    var userStoryCard = ui.helper[0];
    var selectedUserStoryJsonObject =
        dataView.findContext(userStoryCard).dataItem;

    var newX = ui.position.left;
    var newY = ui.position.top;

    Sys.Observer.setValue(selectedUserStoryJsonObject, "X", newX);
    Sys.Observer.setValue(selectedUserStoryJsonObject, "Y", newY);
    dataContext.saveChanges();
}

The first line gets the DOM element that just the user just completed dragging. The second line retrieves the in memory JSON object represented by that DOM element. The next couple of lines get the X and Y that the user story card was dragged to relative to their parent DOM element. And then something odd happens.

Sys.Observer.setValue looks so complicated. Why couldn’t we just call selectedUserStoryJsonObject.X = newX?

Well, if we’d done that the DataView wouldn’t know about the change we just made. If we then tried to call dataContext.saveChanges() nothing would get sent back to the server. Sys.Observer.setValue notifies any interested parties, in this case the DataView, that a value has been changed. And this enables the functionality we saw in part two where the DataView only sends the relevant records back to SharePoint instead of the entire set of JSON objects.

And so, with this code in place, we can now move user story cards around on the page and when we refresh the page or come back days later our layout has persisted back into SharePoint.

Conclusion

In this post you’ve seen how to directly modify the in-memory JSON objects and send them back to SharePoint. Our application is finally getting interesting and I find it quite remarkable how little code it’s taken. In the next post I’ll show how to do master-detail scenario’s for editing user story cards.

Sunday, March 21, 2010

Client Side AJAX Applications in SharePoint 2010 - Part 3

Note: This is one of a multi-part series exploring building client side AJAX applications with ASP.Net AJAX 4 Templating and the WCF Data Services (aka ADO.Net Data Services, aka oData, aka Astoria) in SharePoint 2010.

Part 1 – Exploring WCF Data Services in SharePoint 2010
Part 2 – Creating a read-only page with ASP.Net AJAX 4.0 Templates
Part 3 – Writing data back to SharePoint with ASP.Net AJAX 4.0 Templates
Part 4 - jQuery and Manual JSON Object Manipulation

In the previous two posts in this series I’ve shown how you can access SharePoint data using WCF Data Services and consume that data using ASP.Net AJAX 4 templates. In this post I’ll update the previous example in order to show how to write data back to SharePoint.

Templates and Linking

Before we go into writing data back let’s modify our example to link each item back to the SharePoint DispForm.aspx page. A naïve approach would look like this:

<ul id="userStoriesList" class="sys-template">
       <li><a href="../../Lists/User%20Stories/DispForm.aspx?ID={{ID}}">{{ Title }}</a></li>
</
ul>

It turns out this approach doesn’t work because the DataView won’t look for the data binding syntax inside of attributes. The next approach that might make sense would now be to set href to a data binding syntax and inside there do some string concatenation like so:

<li><a href="{{ '../../Lists/User%20Stories/DispForm.aspx?ID=' + ID }}">{{ Title }}</a></li>

It turns out this doesn’t work either, but for a different reason. Apparently the DataView can’t natively convert some HTML attributes like href and src for reasons Bertrand La Roy describes. The solution is to prefix the attribute with the ASP.Net AJAX system namespace. Typically you would define the system namespace on the HTML body element. But with SharePoint you don’t have access to this element. Fortunately you can declare this namespace on any element so the following code works:

<div xmlns:sys="javascript:Sys">
    <
ul id="userStoriesList" class="sys-template">
        <li><a sys:href="{{ '../../Lists/User%20Stories/DispForm.aspx?ID=' + ID }}">{{ Title }}</a></li>
    </
ul>
</
div>

Great. Let’s move on to updating data.

Live Binding Syntax

The {{ }} template syntax we’ve seen so far is what’s known as one time binding. When you use this syntax the expression is evaluated only once when the template is rendered. What we want is one way live binding and/or two way live binding.

In order to understand the live bindings you need to understand that behind the scenes the DataView keeps an in memory copy of all of the JSON objects it downloads. It's possible for the data in these objects to change. And that's where one way live bindings come in. When you use the syntax { binding [FieldName] }, the DataView will automatically update the binding when the underlying JSON object changes.

Two way live binding happens automatically when you use the one way live binding syntax on HTML INPUT elements. For these scenarios the DataView automatically updates the in memory JSON objects. Then the DataView updates any one way live binding's.

So let’s throw two DataViews onto the page, one with one way live bound hyperlinks, one with two way bound HTML INPUT textbox elements and see what happens:

<script type="text/javascript">
    function pageInit() {
        var dataContext = $create(Sys.Data.AdoNetDataContext, { serviceUri: "/demoprep/_vti_bin/ListData.svc" });

        $create(Sys.UI.DataView,
                {
                    autoFetch: true,
                    dataProvider: dataContext,
                    fetchOperation: "UserStories",
                    fetchParameters: { $top: 20 }
                },
                null,
                null,
                $get("userStoriesList")
            );

        $create(Sys.UI.DataView,
                {
                    autoFetch: true,
                    dataProvider: dataContext,
                    fetchOperation: "UserStories",
                    fetchParameters: { $top: 20 }
                },
                null,
                null,
                $get("userStoriesTable")
            );
    }
    Sys.Application.add_init(pageInit);
</script>

<div xmlns:sys="javascript:Sys">
    <ul id="userStoriesList" class="sys-template">
        <li><a sys:href="{{ '../../Lists/User%20Stories/DispForm.aspx?ID=' + ID }}">{ binding Title }</a></li>
    </ul>

    <table>
        <thead>
            <tr>
                <td>Title</td>
            </tr>
        </thead>
        <tbody id="userStoriesTable" class="sys-template">
            <tr>
                <td><input type="text" sys:value="{binding Title}" /></td>
            </tr>
        </tbody>
    </table>
</div>

Notice the input element with the sys:value bound to title and the new binding syntax in the text of the a href. The result looks like this:

It may not be all that pretty yet, but if you change the text in any of the text boxes the result is immediately updated in the hyperlinks on mouse out. But if you hop back to SharePoint or refresh the page you’ll see that our data hasn’t been updated.

Updating SharePoint, For Real This Time

Just because the DataView updated the in memory JSON objects it’s bound to doesn’t mean it sent a POST request back to ListData.svc. To accomplish that we need to call dataContext.saveChanges(). If you toss in a button below the table like this:

<button onclick="dataContext.saveChanges()">Save Changes</button>

And move the dataContext variable to a higher scope like this:

var dataContext;

function onLoad() {
       dataContext = $create(Sys.Data.AdoNetDataContext, { serviceUri: "/demoprep/_vti_bin/ListData.svc" });

Then when you click the button the AdoNetDataContext figures out which JSON objects changed and sends just those objects back to ListData.svc. How do you know it only sends the rows that changed? If you look at Fiddler there will be a POST to ListData.svc/$batch with something like the following:

--batch_d425-6668-6b05
Content-Type: multipart/mixed;boundary=changeset_8a05-bdcd-0b3e
--changeset_8a05-bdcd-0b3e
Content-Type: application/http
Content-Transfer-Encoding: binary
MERGE http://localhost/demoprep/_vti_bin/ListData.svc/UserStories(3) HTTP/1.1
If-Match: W/"3"
Host: localhost
Accept: application/json
Accept-Charset: utf-8
Content-Type: application/json;charset=utf-8

{
"__metadata":{"uri":"http://localhost/demoprep/_vti_bin/ListData.svc/UserStories(3)",
"etag":"W/\"3\"",
"type":"Microsoft.SharePoint.DataService.UserStoriesItem"},
"ContentTypeID":"0x01080070E2807D369BD94FBD6C057D3110E6D3",
"Title":"Renamed Task",
"Predecessors":{"__deferred":{"uri":"http://localhost/demoprep/_vti_bin/ListData.svc/UserStories(3)/Predecessors"}},
"Priority":{"__deferred":{"uri":"http://localhost/demoprep/_vti_bin/ListData.svc/UserStories(3)/Priority"}},
"PriorityValue":"(2) Normal",
"Status":{"__deferred":{"uri":"http://localhost/demoprep/_vti_bin/ListData.svc/UserStories(3)/Status"}},
"StatusValue":"Not Started",
"Complete":null,
"AssignedToID":null,
"TaskGroupID":null,
"Description":"<div></div>",
"StartDate":"\/Date(1266624000000)\/",
"DueDate":null,
"Points":8,
"Iteration":{"__deferred":{"uri":"http://localhost/demoprep/_vti_bin/ListData.svc/UserStories(3)/Iteration"}},
"IterationID":1,
"ID":3,
"ContentType":"Task",
"Modified":"\/Date(1269128451000)\/",
"Created":"\/Date(1266679964000)\/",
"CreatedByID":1,
"ModifiedByID":1,
"Owshiddenversion":3,
"Version":"3.0",
"Attachments":{"__deferred":{"uri":"http://localhost/demoprep/_vti_bin/ListData.svc/UserStories(3)/Attachments"}},
"Path":"/demoprep/Lists/User Stories"
}
--changeset_8a05-bdcd-0b3e--
--batch_d425-6668-6b05—

And that, as you may have noticed, is the JSON for just a single of the rows that we were displaying.

But how did the AdoNetDataContext know which objects were changed? Jonathan Carter explains this better than I could but basically when the data context downloads JSON objects it injects them with the ability to notify itself when they change. As Jonathan points out it’s a neat trick that would not be possible in a statically typed language.

Summary

We’ve finally gotten to some of the core benefits of using ASP.Net AJAX 4 Templating. With very little JavaScript we were able to cleanly separate our UI from our data access code and write data back to the server. And the amount of HTTP traffic and server operations was kept to a bare minimum, keeping things as fast and responsive as possible. So all we’re missing at this point is how to use it to make something real out of these technologies. And that will be the topic for the next post.

Client Side AJAX Applications in SharePoint 2010 - Part 2

Note: This is one of a multi-part series exploring building client side AJAX applications with ASP.Net AJAX 4 Templating and the WCF Data Services (aka ADO.Net Data Services, aka oData, aka Astoria) in SharePoint 2010.

Part 1 – Exploring WCF Data Services in SharePoint 2010
Part 2 – Creating a read-only page with ASP.Net AJAX 4.0 Templates
Part 3 – Writing data back to SharePoint with ASP.Net AJAX 4.0 Templates
Part 4 - jQuery and Manual JSON Object Manipulation

In Part 1 I described how SharePoint 2010 exposes list data through ListData.svc and WCF Data Services. So far this is interesting, but the next logical question is how would you actually build a Web 2.0 application with the data? One answer is with a new technology called ASP.Net AJAX 4.0 Client Side Templating. I’ll give you some background and then provide a walkthrough of how to build a SharePoint application page that reads and writes data to SharePoint through ListData.svc.

What is ASP.Net AJAX 4.0 Client Side Templating?

In Part 1 I talked about some of the problems with the two existing ways of writing Web 2.0 interfaces. ASP.Net AJAX 4.0 Client Side Templating addresses most of these issues. Specifically it:

  • Minimizes plumbing code
  • Cleanly separates presentation from data access code
  • Simplifies saving data back to the server via Live Data Bindings
  • Has no ViewState!
  • Uses lightweight JSON rather than HTML

Sounds good? Ready to see some code?

Show Me The Code

Note: This walkthrough assumes you haven’t played with Visual Studio 2010 and SharePoint 2010, so please skip ahead to “No Really, Show Me the AJAX Templating Code” if you’ve already seen the sweet SharePoint and Visual Studio integration.

To get started open Visual Studio 2010. Select New Project and for the template navigate to SharePoint then 2010 select “Empty SharePoint Project” (p.s. see Visual Web Part in the list? Make sure you look that up if you haven’t heard of it yet).

If you haven’t been keeping up with what’s new in SharePoint 2010 you should immediately notice that SharePoint integration is built into Visual Studio 2010 out of the box just based on the listed templates. But just wait, it gets much better.

The next screen asks what site you want to use for debugging. It’s asking this because when you hit F5 (“Start Debugging”) Visual Studio 2010 packages up your wsp file, deployes it to the URL you enter here, attaches to the w3wp process, and opens a browser into SharePoint, and you’re immediately in debug mode!

If you aren’t deeply impressed then you haven’t developed in WSS 3.0. Now we aren’t writing any server side code in this tutorial, but it’s still extremely impressive how much work Microsoft has put into making SharePoint a first class citizen.

Once you have your site select Project -> Add New Item and select Application Page.

Visual Studio then plops out a page that’s 100% ready to go including the correct Content controls for the SharePoint master page.

<%@ Assembly Name="$SharePoint.Project.AssemblyFullName$" %>
<%@ Import Namespace="Microsoft.SharePoint.ApplicationPages" %>
<%@ Register Tagprefix="SharePoint" Namespace="Microsoft.SharePoint.WebControls" Assembly="Microsoft.SharePoint, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" %>
<%@ Register Tagprefix="Utilities" Namespace="Microsoft.SharePoint.Utilities" Assembly="Microsoft.SharePoint, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" %>
<%@ Register Tagprefix="asp" Namespace="System.Web.UI" Assembly="System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" %>
<%@ Import Namespace="Microsoft.SharePoint" %>
<%@ Assembly Name="Microsoft.Web.CommandUI, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" %>
<%@ Page Language="C#" AutoEventWireup="true" CodeBehind="ApplicationPage1.aspx.cs" Inherits="MyAppPage.Layouts.MyAppPage.ApplicationPage1" DynamicMasterPageFile="~masterurl/default.master" %>

<asp:Content ID="PageHead" ContentPlaceHolderID="PlaceHolderAdditionalPageHead" runat="server">

</
asp:Content>

<
asp:Content ID="Main" ContentPlaceHolderID="PlaceHolderMain" runat="server">
Hello World!
</asp:Content>

<
asp:Content ID="PageTitle" ContentPlaceHolderID="PlaceHolderPageTitle" runat="server">
Application Page
</asp:Content>

<
asp:Content ID="PageTitleInTitleArea" ContentPlaceHolderID="PlaceHolderPageTitleInTitleArea" runat="server" >
My Application Page
</asp:Content>

(ok, the “Hello World!” is mine). Now if you hit F5 or click Build -> “Deploy Solution” then as I mentioned SharePoint packages up the application page into a wsp file and deployes it out to SharePoint. Now if you navigate to your application page (in my case http://localhost/mysite/_layouts/MyAppPage/ApplicationPage1.aspx) then you should see something like this:

Amazed? You should be. We didn't write a line of evil looking CAML, didn't have to write any DOS batch files, didn't have to deploy to the GAC, we probably didn't even notice that a feature.xml file was created behind the scenes. Yet already we're deployed! Ok, enough exuberance, back to business.

No Really, Show Me the AJAX Templating Code

Before writing code you should probably create some lists to read and write to. As I mentioned in a previous post I like tracking user stories in SharePoint. So for my examples I have an Iterations list and a User Stories list, which is based off of the Tasks content type (automatically giving you a number of columns like Priority).

After creating some lists you need to ensure you have the latest ASP.Net Ajax Library. I suspect an updated copy will be included with the RTM copy of SharePoint 2010, but in my case I downloaded the latest copy and moved the entire Scripts directory into C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\TEMPLATE\LAYOUTS\ajax. Note that out of the box SharePoint 2010 already included some of these files in LAYOUTS, but it was missing others, so this step was necessary.

So here is the absolute simplest templating example that still works:

<asp:Content ID="PageHead" ContentPlaceHolderID="PlaceHolderAdditionalPageHead" runat="server">
    <script type="text/javascript" src="../ajax/MicrosoftAjax.js"></script>
    <script type="text/javascript" src="../ajax/MicrosoftAjaxDataContext.js"></script>
    <script type="text/javascript" src="../ajax/MicrosoftAjaxTemplates.js"></script>
    <script type="text/javascript" src="../ajax/MicrosoftAjaxAdoNet.js"></script>
</
asp:Content>

<
asp:Content ID="Main" ContentPlaceHolderID="PlaceHolderMain" runat="server">
    <script type="text/javascript">
        function pageInit() {
            var dataContext = $create(Sys.Data.AdoNetDataContext, {
                       serviceUri: "/demoprep/_vti_bin/ListData.svc"
                     });

            $create(Sys.UI.DataView,
                {
                    autoFetch: true,
                    dataProvider: dataContext,
                    fetchOperation: "UserStories",
                    fetchParameters: { $top: 20 }
                },
                null,
                null,
                $get("userStoriesList")
            );
        }
        Sys.Application.add_init(pageInit);
    </script>

    <ul id="userStoriesList" class="sys-template">
        <li>{{ Title }}</li>
    </ul>
</
asp:Content>

The four script references in the PageHead Content control are necessary to define the required ASP.Net AJAX JavaScript classes you’ll need for templating. But the Main content control is where things get interesting.

Skip over the JavaScript for now and notice the unordered list named userStoriesList. Inside of the li elements is a strange notation: {{ Title }}. This specifies a binding to the name of a SharePoint field. You could also have put in PriorityValue or any of the other columns specified in the JSON that I showed in Part 1.

As far as the JavaScript, the dataContext variable is a class of type AdoNetDataContext, which knows how to talk to WCF Data Services like ListData.svc. It inherits from a JavaScript class called DataContext which is extensible and knows how to talk with other protocols, but us SharePoint developers probably won’t care about this most of the time.

The userStoriesTemplate variable is a class of type DataView. This is where the magic happens. DataViews pull data from a DataContext, duplicates the innerHtml of an associated DOM element for each row returned (e.g. userStoriesList), and replace the binding syntax with real data. Seems simple enough, but it is actually quite sophisticated when you get to some more advanced scenarios that I’ll describe later.

One last thing to notice is the sys-template class. This is a prespecified CSS class that ASP.Net AJAX sets to display: block when it is finished rendering. It’s a good idea to create a CSS class for sys-template with display: none so that end users don’t see your template code while the page is loading. Obviously you should do this in a separate CSS class in LAYOUTS or if you’re lazy you can put it in the PageHead content control like so:

<STYLE type="text/css">
       .sys-template { display: block; }
</STYLE>

<
asp:Content ID="PageHead" ContentPlaceHolderID="PlaceHolderAdditionalPageHead" runat="server">
       <script type="text/javascript" src="../ajax/MicrosoftAjax.js"></script>
       <script type="text/javascript" src="../ajax/MicrosoftAjaxDataContext.js"></script>
       <script type="text/javascript" src="../ajax/MicrosoftAjaxTemplates.js"></script>
       <script type="text/javascript" src="../ajax/MicrosoftAjaxAdoNet.js"></script>
       <STYLE type="text/css">
              .sys-template { display: block; }
       </STYLE>
</
asp:Content>

So now if you click If you click Build -> Deploy Solution you should see something like this:

Summary

Ok, this isn’t super exciting yet. We certainly could have done this much easier with server side code. But bear with me, this has a lot of potential. For instance, how do you write data back to the server? How do you really make it AJAXy? I’ll explore these questions in Part 3 – Writing data back to SharePoint with ASP.Net AJAX 4.0 Templates and subsequent posts in the series.

Client Side AJAX Applications in SharePoint 2010 - Part 1

If you haven't played with it yet, one of the things you'll immediately notice about SharePoint 2010 is that the UI is extremely AJAX centric. For instance modifying or adding list items is either done in-line with partial page refreshes or via lightbox popups. It's very pleasant to use.

One of the consequences of this Web 2.0 design is that the expectations of SharePoint end users will significantly increase. No longer will long delays and/or full page refreshes in your SharePoint based custom applications be acceptable. End users may not be able to clearly articulate it, but unless as developers we update our techniques to match the new technology our SharePoint based applications will begin to feel stale and will stick out from the rest of SharePoint 2010.

Fortunately Microsoft has stepped up the support it offers Web 2.0 developers in a big way. One of the most compelling features is in how SharePoint exposes all list data via REST/ATOM and JSON. This feature combined with the new templating feature of ASP.Net AJAX 4 allows developers to easily write fast, responsive user interfaces. I will explore these two technologies in the following series:

Part 1 – Exploring WCF Data Services in SharePoint 2010
Part 2 – Creating a read-only page with ASP.Net AJAX 4.0 Templates
Part 3 – Writing data back to SharePoint with ASP.Net AJAX 4.0 Templates
Part 4 - jQuery and Manual JSON Object Manipulation

Why do we need yet another framework?

If you attempt to develop Web 2.0 applications in SharePoint with the techniques you have today there are really only two approaches: ASP.Net AJAX and server side controls or manually coding JavaScript, probably with jQuery or other JavaScript Libraries. If you take the ASP.Net AJAX approach you'll undoubtedly be using an UpdatePanel to get partial page refreshes and then writing ASP.Net code like you always have. There are several problems with this approach:

  • It sends the entire ViewState with every asynchronous request – this can result in slow, unresponsive apps (this can actually sneak up on you once you have a lot of production data as happened to me recently)
  • Even without a ViewState the UpdatePanel transmits large amounts of HTML (rather than lightweight JSON data)
  • It runs through the entire page lifecycle on updates, executing everything on the page even if it isn't relevant to what you're doing
  • ASP.Net AJAX doesn't have the simplicity of an ETag based HTTP caching like with a REST solution
  • Finally SharePoint just doesn't work well with UpdatePanel, I can't tell you how many issues I've had with various controls not working out of the box

And if you take the manual JavaScript rout you are faced with a number of different problems:

  • You have to write a lot of plumbing code
    • You have to write a service to return your data
    • You have to pay particular attention to security and permissions since you're exposing your API's to the world
    • You have to convert your data (ideally JSON) into HTML with probably string concatenation or document.createElement statements
  • Your UI code is not clearly separated from your data access code making your code hard to maintain
  • Saving data back to the server involves even more plumbing code

SharePoint 2010 combined with ASP.Net AJAX 4.0 solves these problems. It probably introduces new problems too, but I haven't run into too many thus far. So over the next couple of posts I hope to convince you that we do need yet another web framework and that Microsoft is headed in a great direction with these technologies.

So How Does it Work?

In WSS 3.0 you were able to access list data with the SOAP based Lists.asmx. This technique worked well for .Net desktop apps or server side application pages, but not so well with client side JavaScript. SharePoint 2010 exposes its list data with REST based ListData.svc (though WCF Data Services which I'll describe in more detail below). Once you install WCF Data Services on a machine with SharePoint 2010 you can simply navigate to http://[server]/[optional site]/_vti_bin/ListData.svc and see all of the lists you have permission to access. The result will look like this:

<?xml version="1.0" encoding="iso-8859-1" standalone="yes"?>
<
service xml:base="http://localhost/mySite/_vti_bin/ListData.svc/"
   
xmlns:atom="http://www.w3.org/2005/Atom"
   
xmlns:app="http://www.w3.org/2007/app"
   
xmlns="http://www.w3.org/2007/app">
  <
workspace>
    <
atom:title>Default</atom:title>
    <
collection href="Iterations">
      <
atom:title>Iterations</atom:title>
    </
collection>
    <
collection href="UserStories">
      <
atom:title>UserStories</atom:title>
    </
collection>
  </
workspace>
</
service>

You can then append the names of lists (e.g. http://localhost/site/_vti_bin/ListData.svc/UserStories) and immediately view the list items in the list as an ATOM feed. A typical result looks like this:

<?xml version="1.0" encoding="iso-8859-1" standalone="yes"?>
<
feed xml:base="http://localhost/mySite/_vti_bin/ListData.svc/"
   
xmlns:d="http://schemas.microsoft.com/ado/2007/08/dataservices"
   
xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata"
   
xmlns="http://www.w3.org/2005/Atom">
  <
title type="text">UserStories</title>
  <
id>http://localhost/mySite/_vti_bin/ListData.svc/UserStories</id>
  <
updated>2010-03-20T02:20:26Z</updated>
  <
link rel="self" title="UserStories" href="UserStories" />
  <
entry m:etag="W/&quot;3&quot;">
    <
id>http://localhost/mySite/_vti_bin/ListData.svc/UserStories(2)</id>
    <
title type="text">Blah</title>
    <
updated>2010-03-19T15:13:28-05:00</updated>
    <
author>
      <
name />
    </
author>
    <
link rel="edit" title="UserStoriesItem" href="UserStories(2)" />
    <
link rel="http://schemas.microsoft.com/ado/2007/08/dataservices/related/Predecessors"
       
type="application/atom+xml;type=feed"
       
title="Predecessors"
       
href="UserStories(2)/Predecessors" />
    <
link rel="http://schemas.microsoft.com/ado/2007/08/dataservices/related/Priority"
       
type="application/atom+xml;type=entry"
       
title="Priority"
       
href="UserStories(2)/Priority" />
    <
link
       
rel="http://schemas.microsoft.com/ado/2007/08/dataservices/related/Status"
       
type="application/atom+xml;type=entry"
       
title="Status"
       
href="UserStories(2)/Status" />
    <
link rel="http://schemas.microsoft.com/ado/2007/08/dataservices/related/Iteration"
       
type="application/atom+xml;type=entry"
       
title="Iteration"
       
href="UserStories(2)/Iteration" />
    <
link rel="http://schemas.microsoft.com/ado/2007/08/dataservices/related/Attachments"
       
type="application/atom+xml;type=feed"
       
title="Attachments"
       
href="UserStories(2)/Attachments" />
    <
category
       
term="Microsoft.SharePoint.DataService.UserStoriesItem"
       
scheme="http://schemas.microsoft.com/ado/2007/08/dataservices/scheme" />
    <
content type="application/xml">
      <
m:properties>
        <
d:ContentTypeID>0x01080070E2807D369BD94FBD6C057D3110E6D3</d:ContentTypeID>
        <
d:Title>Do Something</d:Title>
        <
d:PriorityValue>(2) Normal</d:PriorityValue>
        <
d:StatusValue>Not Started</d:StatusValue>
        <
d:Complete m:type="Edm.Double" m:null="true" />
        <
d:AssignedToID m:type="Edm.Int32" m:null="true" />
        <
d:TaskGroupID m:type="Edm.Int32" m:null="true" />
        <
d:Description>&lt;div&gt;&lt;/div&gt;</d:Description>
        <
d:StartDate m:type="Edm.DateTime">2010-02-20T00:00:00</d:StartDate>
        <
d:DueDate m:type="Edm.DateTime" m:null="true" />
        <
d:Points m:type="Edm.Double">3</d:Points>
        <
d:IterationID m:type="Edm.Int32">3</d:IterationID>
        <
d:ID m:type="Edm.Int32">2</d:ID>
        <
d:ContentType>Task</d:ContentType>
        <
d:Modified m:type="Edm.DateTime">2010-03-19T15:13:28</d:Modified>
        <
d:Created m:type="Edm.DateTime">2010-02-20T15:32:26</d:Created>
        <
d:CreatedByID m:type="Edm.Int32">1</d:CreatedByID>
        <
d:ModifiedByID m:type="Edm.Int32">1</d:ModifiedByID>
        <
d:Owshiddenversion m:type="Edm.Int32">3</d:Owshiddenversion>
        <
d:Version>3.0</d:Version>
        <
d:Path>/mySite/Lists/User Stories</d:Path>
      </
m:properties>
    </
content>
  </
entry>
</
feed>

You can append various parameters to filter, sort, or paginate your data from this page. For example http://localhost/site/_vti_bin/ListData.svc/UserStories?$filter=(PriorityValue eq '(2) Normal')&$orderby=Title will return list items whose priority is normal sorted by title. See Query Expressions on MSDN for more details.

Now if you're interested you can take it a step further by viewing the data as JSON. To do this you'll need to modify the HTTP headers. Fortunately there is a tool from Microsoft for viewing, replaying, and modify HTTP sessions called Fiddler. After you install it open Fiddler from the add-in in your web browser (I used IE, but Firefox should work too). Then refresh the web page for a list (in my case ListData.svc/UserStories). Now in Fiddler drag the last request to the Request Builder, ensure “Automatically Authenticate” is checked in options, and replace “Accept: */*” with “Accept application/json”. When you click Execute you should see something like:

{
"d" : {
"results": [
{
"__metadata": {
"uri": "http://localhost/demoprep/_vti_bin/ListData.svc/UserStories(2)",
"etag": "W/\"3\"",
"type": "Microsoft.SharePoint.DataService.UserStoriesItem"
}, "ContentTypeID": "0x01080070E2807D369BD94FBD6C057D3110E6D3",
"Title": "Blah",
"Predecessors": {
"__deferred": {
"uri": "http://localhost/demoprep/_vti_bin/ListData.svc/UserStories(2)/Predecessors"
}
}, "Priority": {
"__deferred": {
"uri": "http://localhost/demoprep/_vti_bin/ListData.svc/UserStories(2)/Priority"
}
}, "PriorityValue": "(2) Normal", "Status": {
"__deferred": {
"uri": "http://localhost/demoprep/_vti_bin/ListData.svc/UserStories(2)/Status"
}
}, "StatusValue": "Not Started",
"Complete": null,
"AssignedToID": null,
"TaskGroupID": null,
"Description": "<div></div>",
"StartDate": "\/Date(1266624000000)\/",
"DueDate": null,
"Points": 3,
"Iteration": {
"__deferred": {
"uri": "http://localhost/demoprep/_vti_bin/ListData.svc/UserStories(2)/Iteration"
}
}, "IterationID": 3,
"ID": 2,
"ContentType": "Task",
"Modified": "\/Date(1269011608000)\/",
"Created": "\/Date(1266679946000)\/",
"CreatedByID": 1,
"ModifiedByID": 1,
"Owshiddenversion": 3,
"Version": "3.0",
"Attachments": {
"__deferred": {
"uri": "http://localhost/demoprep/_vti_bin/ListData.svc/UserStories(2)/Attachments"
}
}, "Path": "/demoprep/Lists/User Stories"
}
]
}
}

That's SharePoint data as JavaScript Object Notation. Notice the etag and href's to associated list data. Cool.

But ListData.svc isn't just for reading data. To experiment with it try deleting a list item. To do this in fiddler set the HTTP verb to “DELETE”, set the URL to a particular list item's URL (e.g. http://localhost/site/_vti_bin/ListData.svc/UserStories(2) where 2 is the ID for a list item), leave Request Headers empty, and hit Execute. Refresh the list of items and yours is gone. Permanently. The item isn't even in the recycle bin.

Security in WCF Data Services

So at this point you are undoubtedly concerned about security. Just to give some background ListData.svc is provided by a technology called WCF Data Services, which was formerly called ADO.Net Data Services, which was code named Astoria. And WCF Data Services is an implementation of the open data protocol (OData) which Scott Hanselman has an excellent podcast about (no really, download it now and listen to it on your next commute).

This technology was originally designed for exposing database data. And in this scenario you do have to handle your own security. SharePoint 2010, however, takes things a step further by giving you WCF Data Services (nearly) out of the box while also handling your authorization security.

For instance if you modify the row level permissions for a list item to deny read permissions to a user then ListData.svc will not return that list item for that user. If you try to access the URL for that list item directly SharePoint returns a 404. If you try to modify or delete a list item you don't have permissions to modify then SharePoint returns a 401 Unauthorized.

Very cool. I find this especially nice because you don't have to worry about maintaining your security code. It's intimately tied to SharePoint 2010's security model which is already quite sophisticated.

Incidentally, security is not all that's included when you use these services. SharePoint also performs data validation and returns the appropriate HTTP error codes if you for instance violate the awesome new new uniqueness constraint feature of SharePoint 2010.

Summary

SharePoint offers everything you've seen so far (nearly) out of the box and without a line of code. This sets the stage for writing some nice client side applications with technologies like ASP.Net AJAX 4.0 client site templating. Which is exactly what I intend to describe in Part 2 – Creating a read-only page with ASP.Net AJAX 4.0 Templates.