vibrantcode

Full Width Azure Portal for Big Monitors!

andrew@andrewnurse.net — Mon, 29 Apr 2013 00:00:00 -0700

DISCLAIMER: I work on the Azure team, but not on the portal. I think the portal is great, I just prefer a full-width layout. Use at your own risk.

I use the Azure portal A LOT. Since nuget.org is hosted on Azure, that’s probably not a shock :). One thing I don’t like (personal preference, I know some people like it) is that it’s a fixed-width site. I have all this space on my monitor going unused!

You know what would look better? A full width layout:

Fortunately, it’s easy to make it look like that, at least on Chrome. First, download the Stylish extension.

Then, click the new Stylish button on your toolbar and choose “Manage installed styles”

Choose “Write new style” to go to the style editor. Enter the following CSS and mark it to run for all URLs on the domain “manage.windowsazure.com”:

#headerbar-wrapper {
	width: 100%;
	margin-left: 0;
	left: 0;
}
#fxshell-navpane {
	left: 0;
	margin-left: 0;
}
#drawer {
	margin-left: 0;
	left: 0;
	width: 100%;
}
#aux-foreground {
	width: 100%;
	left: 0;
	margin-left: 0;
}

I also put it on a Gist here: https://gist.github.com/anurse/5485124

Save the stylesheet, reload the portal and POOF, instant full-width layout! Enjoy!

Lambdas vs. Method Groups

andrew@andrewnurse.net — Tue, 19 Feb 2013 00:00:00 -0800

Update: Due to a glitch in my code I miscalculated the difference. It has been updated. See full history of the code in the Gist. The outcome is still essentially the same :)

I was talking with David Fowler when he mentioned something I found surprising: “Lambdas are more efficient than Method Groups”. My initial reaction was that he was entirely wrong, but he explained why and then I decided to do some analysis. Here’s what I found.

I wrote a simple program that would pass a method group in to a method in a big loop:

for (int i = 0; i < Iterations; i++)
{
	Call(Dump);
}

And later on, it tried the same with a simple lambda

for (int i = 0; i < Iterations; i++)
{
	Call(s => Dump(s));
}

Where Call takes an Action<string> and Dump takes a single string argument and just does nothing:

private static void Call(Action<string> act)
{
	act("Hi!");
}

private static void Dump(string str)
{
    
}

See the full code for the program as a Gist.

I put in some simple Stopwatch-based timing and Console.ReadLine calls to allow me to advance the app through the steps slowly. The results from the Stopwatch (which timed the whole process and took an average) were interesting on their own. I pre-JIT the methods by calling them outside the loops first, and with 100,000,000 iterations, I got this output:

Running Method Group Test
Finished Method Group Test
Elapsed: 1.5171E+006ns
Average: 1.5171E-002ns
High-Precision? Yes
Press Enter to Continue

Running Lambda Test
Finished Lambda Test
Elapsed: 9.9125E+005ns
Average: 9.9125E-003ns
High-Precision? Yes
Press Enter to End

The lambda case is noticibly faster! Note the difference in exponent (we’re dealing with tiny numbers). Ok, so not much faster, but still, that’s a significant result.

Now, what about memory. So I pulled up perfmon and added the .NET “Allocated Bytes/sec” counter. I’m not sure if that’s the right one, but it certainly seemed to illustrate the point. This is what I saw:

The big spike at the beginning (highlighted in Orange) is the Method Group round. The flat line (highlighted in Green) is the Lambda round. This must be where the problem lies.

Sure enough, looking at the code, we see something interesting. To simplify, I’ve made simple instance methods that just perform the call with a method group and a lambda (see the Gist). Now to decompile them.

.method private hidebysig instance void MethodGroup() cil managed
{
    .maxstack 8
    L_0000: nop 
    L_0001: ldnull 
    L_0002: ldftn void Curious.Program::Dump(string)
    L_0008: newobj instance void [mscorlib]System.Action`1<string>::.ctor(object, native int)
    L_000d: call void Curious.Program::Call(class [mscorlib]System.Action`1<string>)
    L_0012: nop 
    L_0013: ret 
}

First the one with the Method Group (above). Hmm… at L_0008 we see that we’re creating a new delegate and putting the function pointer to Dump in. Make sense. So why doesn’t the lambda version create the same amount of memory?

.method private hidebysig instance void Lambda() cil managed
{
    .maxstack 8
    L_0000: nop 
    L_0001: ldsfld class [mscorlib]System.Action`1<string> Curious.Program::CS$<>9__CachedAnonymousMethodDelegate5
    L_0006: brtrue.s L_001b
    L_0008: ldnull 
    L_0009: ldftn void Curious.Program::<Lambda>b__4(string)
    L_000f: newobj instance void [mscorlib]System.Action`1<string>::.ctor(object, native int)
    L_0014: stsfld class [mscorlib]System.Action`1<string> Curious.Program::CS$<>9__CachedAnonymousMethodDelegate5
    L_0019: br.s L_001b
    L_001b: ldsfld class [mscorlib]System.Action`1<string> Curious.Program::CS$<>9__CachedAnonymousMethodDelegate5
    L_0020: call void Curious.Program::Call(class [mscorlib]System.Action`1<string>)
    L_0025: nop 
    L_0026: ret 
}

Well, it certainly does more. But the key parts start at L_0001. The first thing this code does is check if the “CS$<>9__CachedAnonymousMethodDelegate5” member has a value. If not, it fills it in, at L_0014. But if it DOES have a value, the code just invokes that cached value! BINGO! The Method Group version allocates a new object every single time it is run whereas the lambda version uses an instance (or static, as necessary) field to cache the delegate.

So, what does this mean for you? Well, probably not much. Notice that all these numbers were tiny, so we’re talking about micro-optimizations. In fact, the most important take-away here is to remember that just because something seems faster, doesn’t mean it is faster. In David’s case, he discovered this in SignalR because they have a non-trivial number of lambdas allocated per connection, and there can be a LOT of connections. I know that I’ll probably switch to using Lambdas, just because it doesn’t seem to have any ill effects and if it saves us a few bytes, why not.

Please do note that I’m no performance expert. If you see a problem in this post, please tell me! It’s possible I’ve gotten this completely wrong somehow, so if this kind of performance is crucial to your app, you should definitely do your own profiling! And remember, the only way to make performance gains is to measure measure measure. You can’t just eyeball perf ;).

Look Ma! No GAC! Razor IntelliSense without GACing

andrew@andrewnurse.net — Wed, 06 Jun 2012 00:00:00 -0700

If you’ve done any work with the open-source ASP.Net Web Stack (on Codeplex!) then you may have encountered one annoying issue: If you install an MVC 4 release, your source code version of MVC stops working. This is because our releases put the assemblies in the Global Assembly Cache, which is basically a special folder which overrides the Bin folder of your application. There are many reasons we do this, for example it is required if you are going to NGen an assembly. But specific reasons aside, the fact is that we install our assemblies in the GAC and that isn’t likely to change in “v2” (MVC4/WebPages2/Razor2). So, how can we start working around this problem?

WARNING: This is totally unsupported. I’m happy to help in the comments but if you call Product Support they won’t help you with this!

Step 0: Capture Binaries

Before you go uninstalling stuff, you should do one of these two things:

Grab the assemblies necessary for VS BEFORE uninstalling OR
Be prepared to build new ones from source

If you aren’t prepared to do #2, make sure you do #1! The Assemblies are in C:\Program Files (x86)\Microsoft ASP.Net. Here’s the full list of the ones you need, collect them and copy them into a separate folder (because Step 1 will delete them):

System.Web.Razor.dll
System.Web.WebPages.Razor.dll
System.Web.WebPages.Deployment.dll
System.Web.WebPages.dll
System.Web.Mvc.dll

Step 1: UnGACing

The first step is simply to remove the assemblies from the GAC. We only need to remove the “V2” Assemblies (“V4” for MVC related assemblies) because the GAC’d copy of v1 (v3 for MVC) won’t interfere with a version you build from source. NOTE: This will put Razor IntelliSense and other Visual Studio features relating to MVC in a somewhat broken state, but we’ll fix that in Step 2.

The first thing you might notice if you try to remove our assemblies from the GAC in the “usual” way is that they are locked by Windows Installer:

Ok, that kinda sucks. But fortunately this actually works to our benefit because we can use Windows Installer to remove the assemblies for us! If you installed MVC4/WebPages2 for Visual Studio 2010, this is easy, just go to Add/Remove Programs and uninstall the following two entries: “Microsoft ASP.Net MVC 4 Runtime” and “Microsoft ASP.Net WebPages 2 Runtime”. DO NOT remove the matching Visual Studio tools entries!

If you installed Visual Studio 2012, we don’t actually add our individual components to the Add/Remove Programs list in order to give you just one thing to uninstall (Visual Studio 2012). But, you can still uninstall our components! It just takes a little PowerShell Magic. So, open up an Admin PowerShell prompt (don’t worry, these are all built-in commands so copy-paste away)

First, run this command to uninstall ASP.Net MVC 4:

gwmi Win32_Product | where { $_.Name -eq "Microsoft ASP.Net MVC 4 Runtime" } | ForEach { $msi = Convert-Path "$($_.InstallSource)\*.msi"; msiexec /x $msi }

Click through the prompts and click “Yes” when you are asked to confirm even though 1 or more products will stop working (that’s VS ;)). Then, run a slightly different script to remove ASP.Net Web Pages 2 (which includes Razor):

gwmi Win32_Product | where { $_.Name -eq "Microsoft ASP.Net Web Pages 2 Runtime" } | ForEach { $msi = Convert-Path "$($_.InstallSource)\*.msi"; msiexec /x $msi }

Step 2: Fixing VS.

Now, some of the VS tooling will be broken :(. Pretty much just Razor IntelliSense but that’s one of the best features! (Though I may be biased ;)). VS tries to load System.Web.Razor (and some other friends) in order to parse Razor documents, but in Step 1, we removed it from the GAC and VS doesn’t have a local copy, so it doesn’t know where to find them!

The fix for this is simple: Give VS a local copy! So, first grab the binaries from Step 0 (if you collected them), or build new binaries from our source code (just follow the steps on our CodePlex site and then grab the binaries from the bin folder). Then, copy them to the following folder:

C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\PrivateAssemblies

Of course, replace “Program Files (x86)” with “Program Files” if you’re on a 32-bit OS. And replace “Microsoft Visual Studio 11.0” with “Microsoft Visual Studio 2010” if you’re using VS 2010. Restart VS and you should have IntelliSense! If you run in to issues, try building your project and restarting VS.

Wrapping up

That should be it! Now, please note that this is very unsupported. It works, but if you bork your VS installation, you’re on your own. Ok, not totally on your own, I’m happy to try helping in the comments, but don’t call support because they won’t help you with this!

If you get totally screwed, then you can either: repair VS if you’re using VS 2012, or reinstall MVC 4 if you’re using VS 2010. That should put you back in a known state.

BadImageFormatException - The signature is incorrect

andrew@andrewnurse.net — Fri, 18 May 2012 00:00:00 -0700

Have you ever gotten this exception when running Unit Tests which use Moq?

System.BadImageFormatException : [D:\Path\To\A.dll] The signature is incorrect.

It’s a little strange but it turns out the issue is caused by a missing assembly reference in your Unit Test project. Let’s say you have three projects: Core, Types and Test. In Types, we define a helper interface for writing to the console:

public interface IConsole {
    void WriteLine(string line);
}

In Core, we have code to create and return one:

public class Thingy {
    public virtual IConsole MakeConsole() { ... }
}

And in Test, you have a unit test to test that something in Core never even constructs a console. Tests ONLY references Core, not Types (for whatever reason).

[Fact]
public void DoodadWritesToConsole() {
    // Arrange
    var thingyMock = new Mock<Thingy>();
    var doodad = new Doodad(thingyMock.Object);
    
    // Act
    doodad.DoAThing(writeToConsole: false);
    
    // Assert
    thingyMock.Verify(t => t.MakeConsole(), Times.Never());
}

If you have this set up, you’ll get the BadImageFormatException when running the test (or should, I haven’t verified this particular set of classes but I mocked it up from the code I actually found this in). The problem is that Test doesn’t reference Types, and Types is what provides the interface used as the return value for MakeConsole. All you need to do is add a reference from Test to Types and you’re done!

I basically just wanted to throw up a quick blog post so that I could find it if I encountered this again, but hopefully this also helps a few other folks :)

Yet another blog update!

andrew@andrewnurse.net — Sun, 06 May 2012 00:00:00 -0700

Well, I went and updated my blog again. I got tired of paying for SquareSpace when I was only barely using it. Also, I’m a nerd and nerdy things make me happy. So, I’m trying out GitHub Pages and Jekyll. That means that this entire site is up on GitHub and can be viewed by anyone at any time! The repo is here. It’s all static html that gets run through Jekyll on GitHub’s servers. I have a local copy of Jekyll I use to test things out too. The comments are mostly ported across except for one page (which I’ve contacted Disqus support for). The RSS feed has also moved, but the old one should continue to work. Please resubscribe to the new one though at: http://feeds.feedburner.com/Vibrantcode.

As you can see, I’ve also updated the site layout. It’s a Responsive design as well, which means that the layout adjusts as the browser window size shrinks. It also means it should look pretty good on mobile devices like phones and tablets!

I’ll have more posts later on the details behind the new “engine” but it’s bedtime now ;). Enjoy! Let me know what you think of it!

Shutting down PS-Get

andrew@andrewnurse.net — Thu, 19 Apr 2012 00:00:00 -0700

Hey folks, I wanted to make a quick post to tell you all I'm going to be shutting down work on the PS-Get project. The gallery is being shutdown now and the GitHub Repo will remain open but I do not intend to continue work. There was too much confusion between my project (https://github.com/anurse/ps-get) and PsGet (http://psget.net/). I got a number of people asking me questions about the other and general confusion arose. Also, I'm moving towards a simpler Git-based mechanism for getting my own PowerShell modules (all of which are available at http://github.com/anurse) on to all my machines, so I don't need it anymore.

Don't hesitate to use/view/fork my code, but for now I'm done with the project. It was fun to write, but I can't really justify maintaining it anymore.

gitstorecred - Store HTTPS Credentials for Git

andrew@andrewnurse.net — Mon, 16 Apr 2012 00:00:00 -0700

UPDATE: Thanks to some help from Paul Betts this has been updated to use some new APIs in Git. Check out the blog post on the GitHub Blog for more info!

As you probably know, the ASP.Net team has moved to CodePlex for all of our out-of-band frameworks (MVC, WebPages, WebAPI, etc.). You probably also know that we are using the Git source control system on CodePlex. You may also know that CodePlex only supports the HTTPS transport for Git, which means that a username and password is required every time you push changes to the server:

You can bypass the first prompt by adding your remote with “https://username@server.com/path”, and can even bypass the second by using “https://username:password@server.com/path” as the URL, but that’s a really Bad Idea™. So, like most modern systems, Git probably has a way to store these credentials, right?

Nope.

However, it does have an interesting extension point called “AskPass”. If you set the “core.askpass” config setting, or the GIT_ASKPASS environment variable, to the path to an executable, Git will run that executable every time it needs a username or password. It will pass the prompt it would have written to the console as the command line arguments to this program and it expects the program to write the credential it is requesting (username or password) to the standard output stream. We can use this to our advantage!

GitStoreCred is a small tool I created for doing exactly that. It allows you to replace the Git username and password prompt with a standard Windows credential dialog AND it will store those credentials in the Windows Credential Store (if you check the “Remember” checkbox). Let’s see it in action:

Much better! Once we enter our credentials, everything works and Git pushes happily to the server. Let’s take a look at the Windows Credential Store now (this is in your Control Panel, go to User Accounts then Credential Manager):

See the highlighted entry? That’s my credentials for CodePlex (you’ll see I’ve also got credentials for GitHub there). If I try to push again, git won’t prompt me because gitstorecred just found the existing credential and outputted it.

Enough talking, gimme gimme gimme!

Ok, so you can download gitstorecred from GitHub and you can view the code there too. Just put the exe anywhere and set the GIT_ASKPASS environment variable to it’s full path. I use my PowerShell profile script to do it, since I’m always in PowerShell.

WARNING! HERE BE DRAGONS!

This is super hacky, I literally bashed it up over this weekend. It works, but I make no guarantees. The major known issues are the following:

First, if you type the wrong password, you have to go to Credential Manager (above) and either edit or just remove your credential. Gitstorecred has no way of getting feedback from Git so it doesn’t know that your password was wrong (for example) and will just keep sending the wrong password. If you checked “Remember” then the only way to get it to prompt you again is to remove the credentials.

Second, it doesn’t seem to get in the way of SSH-based pushing, but it might so be careful. If you ever find Git hanging when you run “git push”, just clear out the GIT_ASKPASS setting and try again. It’s entirely possible gitstorecred isn’t responding the way Git expects.

Third, we only store one set of credentials PER SERVER. That means that if you have two user accounts for some reason (say “anurse” and “bnurse”) for some server (say “github.com”), gitstorecred just stores the first one as the credentials for “github.com” and passes them along. Due to some issues with the Windows Credential API I use, it can’t easily key the credential off your user account. For that reason, I highly recommend adding your git remote WITHOUT the user name (i.e. “git remote add https://git01.codeplex.com/aspnetwebstack”). That way gitstorecred will handle the username and password and you won’t get in to weird issues. Of course, the credential store is per windows user, so if you have two separate users on the same machine, they can have different credentials for the same server.

Please do try it out though, the worst that should happen is it won’t work and you’ll have to turn it off. I’m happy to take pull requests to try and make it more powerful/robust (for example, there are better Windows Credential APIs I could use but they are harder to access from Managed Code). With a tool like this, and the fact that Git is much smarter about HTTP pushing now, I’m switching almost exclusively to HTTPS for my Git pushes.

Enjoy! Here’s the links again: Download EXE, Source Code on GitHub.

What else is new in Razor v2?

andrew@andrewnurse.net — Fri, 13 Apr 2012 00:00:00 -0700

In my last post I discusses some of the big new features in Razor v2. In this post I'm going to talk a bit about some of the other (admittedly smaller) new features. So, let's get right to it!

Void Elements

The HTML spec defines a certain type of element called a "Void Element" as

[An] element whose content model never allows it to have contents under any circumstance
- W3C HTML 5 Spec section 4.3

Put more simply, a void element is a type of element that can NEVER EVER have contents. It can come in three different forms:

  <!-- Self-closing -->
  <input name="foo" /><p>A different tag</p>
  <!-- Closed -->
  <input name="foo"></input><p>A different tag</p>
  <!-- Unclosed -->
  <input name="foo"><p>A different tag</p>

Any of the following is considered valid HTML5. However, in Razor v1, we only allowed the first two, because we had a much simpler parser. In Razor v2, you can now use the third form as well. This works because if a void element's start tag is not self-closed AND is not IMMEDIATELY followed by an end tag (whitespace is allowed) then it is considered closed at the ">" of the start tag. So in Razor, when we parse a void element and reach the ">", we look ahead and check if we see "</[tagname]>" (we allow whitespace between the start and end tags). If we do NOT see it, we consider the tag closed. So this means that if you typed the following inside a code block (i.e. @if() {})

<p><input name="foo">Some content</input></p>

Razor would end the markup block at the "</input>" tag. Why? Because the input element was closed by the ">" of it's start tag, so the "</input>" tag is considered an extraneous end tag. Since it has no matching start tag (remember the "<input>" is already closed), we think that it belongs to a start tag Razor can't see (because it's outside of the code block this markup is within or even in a different document) and we end the block there.

For the most part, this shouldn't affect you adversely, since syntax like the sample above is invalid HTML, but please let me know if you end up in an edge case where this is occurring in legal HTML.

Finally, what are the elements HTML5 considers void? The spec lists them off for us and Razor uses this exact same list:

area base br col command embed hr img input keygen link meta param source track wbr
- W3C HTML 5 Spec section 4.3

Syntax Tree and Internals Overhaul

NOTE: This part is going to dive in to parser internals a bit. Feel free to skim ;)

The last thing I'm going to talk about is a behind-the-scenes change that's mostly relevant to people hosting the Razor parser. In order to support these exciting new features, we had to overhaul our parser internals and syntax tree structure. Razor now uses full HTML, C# and VB tokenizers and the parse tree gives you access to that granularity. For example, given the code:

@foo.bar.baz

In Razor v1, this would be two Spans (a type of parse tree node), one for "@" and one for "foo.bar.baz". Each span would contain the string pulled from the input file. However, in Razor v2, we produce the same two spans, but now each span is a collection of Symbols. In this case, the first Span contains one symbol "@" and the second Span contains 5 symbols ("foo", ".", "bar", ".", "baz"). This allows us to perform more advanced analysis of the input document without have to reparse strings over and over.

We also reworked our higher-level Syntax Tree nodes, Span and Block. In Razor v1, we broke the file into chunks called spans and used sub-classes of the Span class to mark their type (for example: MarkupSpan, CodeSpan, HelperHeaderSpan, TransitionSpan). In v2, we removed all of those sub-classes, moving to an "Annotations" model. Going back to our previous example, in Razor v1 we would have produced a TransitionSpan and a ImplicitExpressionSpan (a kind of CodeSpan). In v2, both are concrete instances of Span, however they have various properties which attach annotations to control how they behave. For example, each Span has a CodeGenerator annotation which indicates how we generate C#/VB code from the node. They also have an EditHandler annotation which indicates how the editor should behave around this Span. By doing this, we (and even you if you want!) can add new syntax without having to dramatically overhaul all the various pieces of our infrastructure.

In future posts, I'm going to use some of this information to show you how to create a new kind of directive that works at runtime AND design-time (i.e. in the Editor).

Conclusion

Well, that's basically it for Razor v2. It's not a super-long list but believe me it was a lot of work. In many ways, Razor v1 was our "hey, check out this cool language!" release. Razor v2 has primarily been about internal clean-up and future-proofing (not that one can ever be totally future-proof). We added some cool little features, but the work done to the underlying structures should make it easier for us to add even more features and extensibility points in the future.

Enjoy Razor 2, Web Pags 2 and MVC 4!

What's new in Razor v2

andrew@andrewnurse.net — Tue, 10 Apr 2012 00:00:00 -0700

I know, it’s been too long since I blogged, and I won’t even bother saying I’ll try to blog more (though I will) because hopefully you’ll see that for yourself. A lot has happened since I last blogged but the biggest thing is that the ASP.Net Web Stack (which is what we call the out-of-band stuff we ship, like MVC, WebPages, WebAPI and Razor) is now truly an open-source project! We’re hosted on CodePlex (http://aspnetwebstack.codeplex.com/) and if you want more details, check out Scott Hanselman’s awesome post about it.

Now that we’re open, I’ll be able to start talking a little more directly about what we’re doing. It’s very exciting for us, and to alleviate your worries that open-source means Microsoft isn’t working on it anymore, we’re still working full-time on our current release and even starting a bit of planning for v.Next.

All that aside, we also released a new version of Razor since I’ve blogged and I thought I’d give you a quick run-through of the features. In later posts, I’ll go over the details of what’s changed as well as some new information for those hosting Razor outside of ASP.Net on how to take advantage of those features.

~/ – Url resolution made easy, goodbye @Href/@Url.Content!

One of the most common patterns in MVC Views is this:

<script src=”@Url.Content(“~/Scripts/myscript.js”)”></script>

Well, in Razor v2, you can express the same intent with much less code, and it looks much cleaner too:

<script src=”~/Scripts/myscript.js”></script>

Note the lack of any code transitions! How does this work? It’s quite a simple algorithm: Whenever we see an attribute value that starts “~/”, we treat it like a URL and replace it with a call to @Href (in WebPages) or @Url.Content (in MVC). Note that this is ANY attribute, so if you typed the following:

<div class=”~/Foo”>

We’ll treat “~/Foo” as a URL. We made this choice because we didn’t want to limit you to a list of attributes that we think have URLs. Especially when you might want to put URLs in a data- attribute. Imagine using a data- attribute to tell your client-side javascript what the root of your application is:

<html data-root=”~/”>

Then you can use jQuery to access this data: $(document).data(‘root’) and use it when making Ajax calls to make sure your app is portable even if it’s in a sub-folder.

What if you want to inject code snippets in to your URL? Well that’s easy, just treat it like you would if you were injecting code in to other attribute values:

<a href=”~/MySite/@Foo/Bar/@Baz”>Something!</a>

I should note that we’re actually not doing much special here, the code above is equivalent to the following MVC code in Razor v1:

<a href=”@Url.Content(“~/MySite/”)@Foo/Bar/@Baz”>Something!</a>

So we are just resolving that first portion of the URL and then going back to regular string concatenation for the rest.

Conditional Attributes

The other major feature (there are a few others I’ll go over in later posts but they are smaller) is Conditional Attributes. I’ll freely admit we borrowed this feature heavily from the fantastic Spark View Engine written by our very own Louis Dejardins.

Have you ever typed code like this in Razor v1?

@{ var cls = GetClass(); }
<div id=”foo”@if(cls != null) { <text>class = “@cls”</text> }>

If not, let me explain why you’d want to do this. In Razor v1, null was treated the same as an empty string so if you were to have code like this:

<div id=”foo” class=”@cls”>

Then if cls was null, we’d render

<div id=”foo” class=””>

Eugh! That looks ugly! In Razor v2, the same code would render this:

<div id=”foo”>

Note the missing class attribute? We’ve even taken away the leading space! Another feature of this is that we’ll also collapse whitespace within the attribute value:

@{ string foo = null; string bar = “bar” }
<div id=”foo” class=”@foo @bar”>

Becomes:

<div id=”foo” class=”bar”>

We also special case boolean values. If the expression evaluates to false, we treat it the same as null. If it evaluates to true, we render out the attribute name again. This allows you to write code like the following for checkboxes:

<input type=”checkbox” checked=”@isChecked”>

If isChecked is true, we render checked=”checked”, if it’s false, we don’t render the checked attribute at all.

Finally, we do NOT treat String.Empty ("") like null in this case. If the expression evaluates to an empty string, we WILL render the attribute:

@{ var foo = ""; }
<div class=”@foo”>

Renders:

<div class=””>

The reason for this lies in the difference between null and String.Empty. Null indicates the complete absence of a value, whereas String.Empty is a value, a string of length 0.

In the currently released (Beta) version of Razor, we do this for all attributes. However, in the next release (and, in fact, in the live code on CodePlex) we have entirely disabled the conditional attributes feature for data- attributes. You'll still get the ~/ URL resolution, but we won't remove null values or do fancy tricks with booleans in data- attributes. This was done because those attributes are usually read from JavaScript, so the semantics are different and we wanted to give you full control.

There’s much more!

That’s just a quick summary. I’ll be publishing another post soon with even more new features. Then, we’ll go in to how Conditional Attributes is implemented and how you can make sure your custom Razor hosts support this feature.

Please don’t hesitate to let me know what you think in the comments!

PS-Get 0.6 - Now with a Gallery!

andrew@andrewnurse.net — Thu, 22 Dec 2011 00:00:00 -0800

I've been hard at work, and not just at my day job! It's been a while but I've finally gotten together and created a proper website for PS-Get including?.(drumroll please)?. a Gallery! ~~Blatantly ripped off~~ Borrowed from the NuGet Gallery Code, the PS-Get Gallery is a place to upload PS-Get modules for others to download. It also contains detailed installation instructions, but thanks to a new installer script, it's a pretty simple install! To install, just run this command from a PowerShell 3.0 or PowerShell 2.0 (with .NET 4.0 workaround, see below) window:

(new-object Net.WebClient).DownloadString("http://install.psget.org")|iex

And then you're all set! There's rudimentary support for automatically injecting the proper code into your Profile, but if you want to do it manually, all you need is Import-Module PS-Get.

So, what's new in PS-Get 0.6?

Import-Package Cmdlet

The Import-Package cmdlet is a one-step cmdlet that takes the name of a module as it's primary argument. It checks if that module is installed, and if not it installs it for you. Once the module is installed, it imports it in to the current environment. For example, consider the following code:

Import-Package PS-Json

If PS-Json is already installed, this is identical to Import-Module PS-Json. However, if PS-Json is not installed, it's install first. This means that if you sync your profile across multiple machines, all you need is some code to check for and install PS-Get and a bunch of Import-Package statements to get all your Modules installed and ready! In fact, the PS-Get install script (which you can download here) takes parameters that let you skip the entire wizard. So you could have code like the following in your profile and have PS-Get and all your favourite packages installed automatically!

if(@(Get-Module PS-Get).Count -eq 0) {
    Write-Host "This is the first time you've run this profile on this machine so we need to install PS-Get first..."
    (new-object Net.WebClient).DownloadString("http://install.psget.org")|iex
}
Import-Module PS-Get
Import-Package PS-MagicStuffThatILove

Package Source Management

New in PS-Get 0.6 is the ability to manage multiple package sources. Using Add-/Remove-/Get-PackageSource you can manage package sources at three scope levels: Machine (global to all users), User (global to all PowerShell sessions for the current user), Session (specific to just this PowerShell session). These sources will all be searched for packages when Install-Package and Update-Package are used to get new packages. The installer script automatically sets up the PS-Get Gallery as a machine-level source. You can still override the source by using the "-Source" parameter of most cmdlets.

Export-Module Cmdlet

PS-Get 0.6 adds a new cmdlet "Export-Module" which can be used to take an existing PowerShell Module and export it to a NuPkg file which can then be uploaded to the PS-Get Gallery. If the module directory contains a NuSpec file, we'll use that, otherwise will gather as much data as we can from the PSD1 manifest file (if there is one). Missing data can be filled in using arguments to the Export-Module cmdlet. Eventually, the plan is to make it easier to manage extra PS-Get-related metadata for Modules, either by augmenting the existing PSD1 manifests or by providing tools to manage NuSpec files for modules.

One thing this is missing right now is dependencies. If your module depends on another module, you'll need to create a NuSpec file for it. Removing this restriction is priority 1 for enhancing the Export-Module cmdlet.

Conclusion/The Future

I hope you'll try it out and start submitting packages to the gallery. It's ready to go, but I'll be perfectly honest that I'm doing this with no budget so I'm working on free/cheap hosting solutions and the site may go down.

If you want to help with the future of PS-Get, check out our GitHub page. Fork the code, file some issues on our tracker or just read the wiki!

Catharsis and Humanity

andrew@andrewnurse.net — Mon, 02 May 2011 00:00:00 -0700

This is a rare non-tech-related blog post, and I hope you'll forgive me for it. It's also possible you may find it controversial, so I'd ask your patience in reading the entire thing before jumping to conclusions. There will likely be another non-tech post soon, since the Canadian Federal Election is wrapping up, but I needed to get this post out before then.

As everyone who isn't living under a rock knows, on Monday, President Obama announced that US Special Forces had located and killed Osama Bin Laden. My girlfriend (who has posted something about this herself) brought up a quotation (attributed to either Mark Twain or Clarence Darrow depending on who you ask) which I think accurately sums up my feelings: "I've never wished a man dead, but I have read some obituaries with great pleasure". There's no question in my mind that Osama Bin Laden was responsible for terrible things, and his death has brought closure to many people. I certainly don't fault the people in Washington DC and New York City for feeling a sudden urge to celebrate out of catharsis. One of the people I follow on Twitter, Stephen Toulouse, tweeted that "catharsis doesn't choose a time to be appropriate" which really makes a lot of sense to me.

Despite all that, I feel? uneasy (for a significant lack of a better term)? about the way we are celebrating the death of another human being, as evil as he was (whatever "evil" truly is). I know that on September 11, 2001, there were people cheering in the streets for the death of human beings too, and I think it's important to remember that as we go through this catharsis. Please do NOT get me wrong, I can fully understand the need to celebrate, it is human to have such a strong outpouring of relief and emotion when closure is found. For the people around the world who were directly affected by the events of 9/11, this will most definitely be an almost happy occasion, and I would not fault them for feeling that way. As I noted earlier, I myself am certainly not upset that Bin Laden is dead or even that people are celebrating that fact.

I think it is entirely natural to feel relieved and to want to express that relief, but now that the catharsis is over, we should rise above our enemies and remember that we respect all life, even those who feel so oppressed as to need to strike out violently. What Osama Bin Laden and al-Qaeda did 10 years ago was WRONG, full stop. However my moral code also tells me that any time a human being is killed by another, it is because of a failure in our society. I am still struggling to figure out what that means, since I too feel a lot of relief over the closing of this chapter in our history, but I think what's most important is that we recognize this event for what it truly is: A purge of our emotions and frustrations which have been building up since 9/11, not a victory for humanity as a whole. There's nothing wrong with celebrating the former, as long as you keep the latter in mind and reflect on what we need to do in order to achieve true peace.

Please don't hesitate to post legitimate responses and discussions to this post below. I welcome disagreements. However, I will not hesitate to remove hurtful or hateful comments. Above all else, RESPECT each other, because that is how we rise above.

P.S. Apparently "catharsis" is my word of the day

Meme Helper for ASP.Net WebPages

andrew@andrewnurse.net — Fri, 01 Apr 2011 00:00:00 -0700

We're working hard on the next versions of ASP.Net WebPages, MVC and Razor, but in the mean time, we wanted to preview some of the work we're doing for the next version. So, I'm releasing one of the new helpers we're working on as an early preview via NuGet.

Have you ever felt that your site was missing an important aspect of web culture? Like an image of a cat, or an embedded 1980s music video? That's what we're hoping to solve with this new helper for ASP.Net WebPages.

The Meme Helper is a simple helper which allows you to easily inject pure unfiltered AWESOME into your site with a single line of code. For example, let's say you want to embed a certain newly released music video that's garnered a lot of interest recently. You could go find the video and add a bunch of HTML embed code from your favorite video sharing site. Or you could use HTML5. Or even better, you could use the MemeHelpers and do it in one line:

@Friday.Embed(type: Friday.FridayType.Normal)

Or maybe you want to reference a certain weather phenomenon which occurs not once, but TWICE in the sky at the same time:

@Rainbow.Embed(type: Rainbow.RainbowType.Double)

Or perhaps you've tired of this whole Meme phenomenon and want to express your exhaustion:

@Facepalm.GetHtml(type: Facepalm.FacepalmType.Picard, height: 200)

If there's a meme, there's probably a helper for it. And if not, you can contribute your own by forking the BitBucket repository!

You can get the MemeHelpers using NuGet (PackageId: FourOne.Memes). Plus, I've attached a Sample Website to this post. Install the package and explore the Memes!!

Oh and remember: Yesterday was Thursday! Thursday! Today it is Friday! Friday! Fun! Fun! Fun! Fun!

DISCLAIMER: This is not actually related to Microsoft in any way. Microsoft does not endorse any of these Memes. This is just the idle rambling of a developer who likes Memes and a certain day in April ;)

New Blog Design and Hoster!

andrew@andrewnurse.net — Sat, 12 Mar 2011 00:00:00 -0800

I decided I was tired of running my blog off of my own server, so I've moved to a hosted blog platform, and redesigned by blog a little as well . The new blog is, well, right here! It's hosted on SquareSpace which is a high-scale blog hosting platform. You might ask why I'm not using an ASP.Net blog system, well the answer is simple: I'm not in the business of running a blog. I hope you enjoy my blog, but I don't feel a need to invest a lot of time and effort into running a blog myself when I can pay someone else to do it! After some review of the options, this ended up being the best one of those I reviewed. You'll notice there's a new design, which is one of the reasons I chose SquareSpace, they have a bunch of really nice templates but more than that, they provide easy customization tools! With only a little bit of CSS, I converted one of their stock themes into something that suited me a little more. All of my previous post should be here, but there were little content glitches during the transfer process. Everything should be there, but some of the older posts might not look as nice

In the same vein, I ported all of the comments over to Disqus, and all future comments will be handled through that service. Again, all the comments should be there, but don't take it personally if your comment has disappeared! If you encounter any content problems in the blog or comments please let me know!

I've also normalized everything to point at the same domain name: http://vibrantcode.com. All the old links should work, but again, let me know if you encounter one that doesn't.

The new blog also uses jQuery Lightbox and the latest version of SyntaxHighlighter.js to give it a little more pop. I hope you like it!

PsGet v0.2 - Now with more awesome! (and fewer version numbers)

andrew@andrewnurse.net — Thu, 17 Feb 2011 00:00:00 -0800

I've just pushed a new version of PsGet out and it has a number of cool new features!

No more version numbers in Module names!

Thanks to both a private patched build of NuGet.exe (this is a temporary fix) and a patch to the main version of NuGet (http://nuget.codeplex.com/SourceControl/changeset/changes/81c1815fd452 - this is the more permanent fix), PsGet's Install-PSPackage script no longer adds the version number to the end of the Module name. So you can name your package files "Foo.psd1" instead of "Foo.0.1.0.0.psd1".

No more nested module path

PsGet no longer uses it's own nested module path (the packages folder under the PsGet install path). It now installs modules to the same place PsGet itself was installed. The plan for the future is to make install destination settable via a parameter to Install-PSPackage, but for now this seemed like a better default.

Totally Self-Hosted

This version of PsGet, unlike the previous version, has no bootstrapper module and is actually a PsGet package itself (it's even on the NuGet gallery: http://nuget.org/Packages/Packages/Details/PS-Get-0-2-0-0). The install script (see next section) handles install it though, so there should be no need to download it yourself.

Install Script

PsGet now has a bootstrapper install script. Unfortunately there are still more hoops to jump through than I'd like, but it's easier to use than the first version. If you're upgrading from v0.1, go ahead and delete your PsGetBoot folder, it's the easiest way to upgrade. From v0.2 on, upgrades will be handled through PsGet itself!

Here's how to install PsGet v0.2. First, go to our download site and download Install-PsGet.ps1:

Once downloaded, you need to temporarily allow PowerShell to run unsigned remote scripts. Open an Admin PowerShell prompt and type the following command:

Set-ExecutionPolicy Unrestricted

Then, run the Install-PsGet.ps1 script you just downloaded. You may get a security prompt, go ahead and accept it. Feel free to skim the contents of the file before you run it. The script will download some files, but it will clean them up when it's finished.' You will be prompted to choose where to install PsGet. The installer scans your PSModulePath environment variable and finds all the possible entries. Pick one of these entries using a number in the list:

The script will download a version of NuGet.exe from my site and then use it to install PS-Get. Since PS-Get includes it's own copy of NuGet.exe, the script will then delete this temporary version when it's finished.

When the script completes, unless you really want to walk on the wild side, make sure you set your execution policy back to the default! If you've never modified it before, that means you should run the following command to get it back to normal:

Set-ExecutionPolicy Restricted

You've now installed PS-Get 0.2! Now, you still have to add it to your profile, just as before (though it has a slightly different name now):

And you're done! Enjoy and please let me know if you have comments!

NuGet + PowerShell = (also) Crazy Delicious

andrew@andrewnurse.net — Fri, 11 Feb 2011 00:00:00 -0800

First off, I should note that I'm kinda paraphrasing this title from Scott Hanselman's excellent PDC 2010 talk, but I'm sure he's ok with that. By now, if you've been following the news coming out of the Microsoft Web Platform and Tools team, you've probably heard about our latest open-source project: NuGet. I'll let Scott Guthrie fill you in if you haven't heard about it already. Well, since shipping Web Pages v1/MVC v3, we've been working on general clean-up of both our code and our processes. For me, part of that entailed wiping all my dev machines and re-installing from scratch (there's a certain level of happiness provided by a nice fresh machine ). As part of that, I was organizing my little PowerShell utility scripts into Modules. I've also been thinking of organizing them a bit so that they could be released, but I ran in to a problem: There isn't really a great way to distribute PowerShell Modules!

In PowerShell, a Module is really just a script within a special folder. When you "Import" the module, the script is run and the functions/variables/etc. it exports are added to the shell environment. There's much more to it, but I'll leave it to MSDN to explain the rest if you're interested.

Deploying a Module is as simple as dropping a folder in C:\Users\[user name]\Documents\WindowsPowerShell\Modules. But in order to get a Module from the Web, like say the very useful PowerShell Community Extensions package, you have to download a ZIP and extract it to the right place. Earlier this week I had a brainwave: "Isn't this _exactly_ what NuGet is designed to do?"

So, I spent a bit of spare time working on something I call "PsGet", which I'm releasing a very early version of right now. PsGet is a PowerShell Module that uses NuGet to deploy other PowerShell Modules. Right now, you do have to manually install PsGet, but I'm hoping to set up a single bootstrapping script that you can download and run to install it automatically. Let's walk through how it works:

First, download the PsGetBoot module from the BitBucket site I've set up. Here's a direct link to the ZIP file. Then, make sure the ZIP is unblocked in Explorer:

Now, open/create your PowerShell Modules folder in Explorer. It needs to go in "C:\Users\[username]\Documents\WindowsPowerShell\Modules". If you have moved your My Documents folder (like I have, to the D: drive in this case), you should update the path accordingly. Open the ZIP file in Explorer and copy the PsGetBoot folder into your Modules folder:

Next, you need to ensure you've enabled the execution of unsigned scripts. If you've already done this, skip this step. Start a PowerShell window as Administrator and run the following command: "Set-ExecutionPolicy RemoteSigned". Enter "Y" at the confirmation prompt.

Ok, if you skipped the last step, open a PowerShell window. Otherwise, you've already got one open . Now we need to open your profile script and add a line to activate PsGet. The profile script is the script that runs every time you open PowerShell. First, let's make sure all the necessary folders exist by running:

mkdir (Split-Path -Parent $profile)

If you get an error saying the item already exists, don't worry, it just means you didn't need to do this step . Next, open your profile script up in notepad (or your favorite text editor) and make sure to create it if it doesn't already exist:

notepad $profile

Type the following somewhere in the main body of your profile. It can go at the end, the beginning, or the middle, but not within a function:

Import-Module PsGetBoot

Then save the file, shutdown your PowerShell windows and open up a new one (Admin privileges are NOT required this time). The line we added to the PowerShell Profile will load in the PsGet "Bootstrapper" module. This is a module which just imports the "PS-Get" module and starts up PsGet. The purpose of this bootstrapper is to allow PsGet to update itself using PsGet .

Once you've started PowerShell back up, you're all done! You now have two new commands. The first is "Get-PSPackage". I should warn you that this command is extremely simple right now and just lists every NuGet package from the main nuget.org feed, so be prepared for it to take a few minutes when you run it. PsGet automatically works against the main NuGet feed at nuget.org, but it doesn't really make sense to install packages like Entity Framework in to PowerShell. So, as a convention for now, packages on nuget.org which are PowerShell modules should have names which start "PS-". For example, I've already posted a very very simple package called PS-Bitz which is a library of useful functions (and by library, I mean a single function, for now ). To find all the PowerShell packages, just go to nuget.org and search for "PS-", or just click here. Once you've found a package, just use the "Install-PSPackage" command to install it:

Install-PSPackage PS-Bitz

Once the package is installed, you should be able to see it when you get the list of available PowerShell Modules:

(I'm not entirely sure why multiple entries show up, please let me know if you know why ).

Now that it's installed, you can add it to your CURRENT shell by running Import-Module:

Import-Module PS-Bitz.0.1.0.0

You can then use all the functions provided by it. In this package, there's a single function used to get the path of special folders like MyDocuments:

If you want to make it available every time you launch PowerShell, just add an "Import-Module" line for it to your Profile, just as we did with PsGetBoot.

As you can probably see, PsGet is pretty rough around the edges right now, but I'm hoping to improve it over time. I've set up a BitBucket repository here, so feel free to fork it and hack away. I'm licensing it under the Apache Public License, which matches NuGet's license. Let me know what you think! And start posting PowerShell module packages to NuGet.org!

Hosting Razor outside of ASP.Net (Revised for MVC3 RC!)

andrew@andrewnurse.net — Tue, 16 Nov 2010 00:00:00 -0800

We recently released the latest preview release of MVC 3, including an updated version of Razor. One of the things we did in this release is simplify the hosting APIs dramatically. I did a demo of these new APIs in a pre-recorded PDC10 talk I did with Scott Hunter, which is available to stream here: http://bit.ly/ac7B0P. As I promised in that talk (and a few others before and after it ), I'm finally blogging about the sample I showed in that talk!

The hosting APIs center around two classes:

RazorTemplateEngine: the primary entry point to the Razor engine
RazorEngineHost: defines the environment in which the generated Razor code will run

First, let's look at the Host. RazorEngineHost is a class with a number of useful properties on it that can tweak the way the generated code behaves. More interestingly (though also more complicated and a topic for later), it can be subclassed and used to swap out the parser and code generator with your own custom code to add new keywords and features to Razor for your scenario! In the sample I showed at PDC, we just used the properties to configure things, like so:

// Set up the hosting environment

// a. Use the C# language (you could detect this based on the file extension if you want to)
RazorEngineHost host = new RazorEngineHost(new CSharpRazorCodeLanguage());

// b. Set the base class
host.DefaultBaseClass = typeof(TemplateBase).FullName;

// c. Set the output namespace and type name
host.DefaultNamespace = "RazorOutput";
host.DefaultClassName = "Template";

// d. Add default imports
host.NamespaceImports.Add("System");
host.NamespaceImports.Add("System.IO");

This code is just doing fairly simple (but powerful) stuff. It first sets the base class for the template. Since we are running Razor outside of ASP.Net, you don't have to use our base classes. In fact, you have to provide your own! The sample contains a simple base class which just writes the template content to a StringBuilder we can retrieve later.

Razor generated code using a certain convention (that can be overridden if necessary). The generated template will contain a public method called "Execute" and that method will contain the users code and calls to one of the following methods:

Write() - When a user types "@foo", we generate Write(foo) in the generated code
WriteLiteral() - When a user types literal markup, like "<p>Foo</p>", we generate WriteLiteral("<p>Foo</p>") in the generated code

The names of these methods can be configured using the GeneratedClassContext property of the host, but we won't go into details on that here. If you want to enable features like Sections, Templates and Helpers in your custom Razor host, you need to set some properties here but we'll cover that another time.

The host also defines the namespace and class name of the generated C# or VB code, and the list of namespaces to be imported into the template's generated code file. Once we've configured our host, we construct a RazorTemplateEngine and give it this host to use:

// Create the template engine using this host
return new RazorTemplateEngine(host);

From there, we're set to generate some code! Let's look at what happens when you click the "Load Template" button in the sample.

// Generate code for the template
GeneratorResults razorResult = null;
using (TextReader rdr = new StringReader(templateTextBox.Text))
{
    razorResult = _engine.GenerateCode(rdr);
}

Now that we have our engine configured, all we need to do is take the template source code (from a text box in this case), open up a TextReader for it, and pass it in to the RazorTemplateEngine we constructed earlier. The result of that call is a structure called GeneratorResults which contains the following properties:

Success - A boolean indicating if parsing and code generation were successful
GeneratedCode - A CodeDOM tree representing the code that the Razor engine generated
ParserErrors - A list of parser errors, if any
Document - The root node of the Razor parse tree (we'll cover that later)

From this one call, you get all that data! You can party on the parse tree, or use the CodeDOM APIs in the core .Net Framework to generate source code. In the sample, I take the CodeDOM tree, compile an assembly using it, then load the assembly and the generated template type. It takes quite a bit of code, so I'll leave that to the sample rather than posting it all here, but it's easily wrapped up in a helper method if you are going to be doing a lot of compiling!

It is important to note that Razor is not an interpreted language. It actually follows the same compilation model as ASPX, despite the drastically different syntax. The first time a user requests a Razor page, we generate the C#/VB code for it and compile it into an assembly. All future requests, until the original file is changed, go against that compiled C#/VB code. This means we can't easily do things like implementing "eval" because we aren't actually running the Razor code directly, we're just using it to generate C#/VB code.

Hopefully that gives you a quick summary of how to get your feet wet hosting Razor outside of ASP.Net.

Also, in MVC RC, we released Visual Studio support for Razor. It's still an early release, so there might be some issues. We'd like to hear from you if you encounter any issues! Email razorvs@microsoft.com with your issues (related to VS tooling only please). If you have other questions about Razor in general, tweet me at @anurse or email me at andrew AT andrewnurse DOT net!

Here's the sample: HtmlGen.zip (65.04 KB) (NOTE: MVC 3 RC is required).

Microsoft.Data - It's not as evil as you think

andrew@andrewnurse.net — Wed, 04 Aug 2010 00:00:00 -0700

I wanted to jump in with my $0.02 (Canadian ) on Microsoft.Data. David Fowler, fellow ASP.Net team member, posted about it earlier today, and the response has been ... active . The message behind most of these responses has been that it encourages bad practices to novice developers. I think there's an important point that's being missed here: It doesn't matter how hard we work, as professional developers, to create clean architectures and abstractions, there's a whole world of novice developers who just don't care about that. They want to write code now and be done with it. This is the audience targeted by Microsoft.Data, and the WebMatrix product as a whole. If novice developers come to the Microsoft platform and see these complicated architectures (which, don't get me wrong, have massive benefits for professional development), they simply won't adopt them, they'll just head to a different platform which allows them to get their job done quickly.

Microsoft.Data is, in my opinion, actually a step in the right direction. For example, let's say I'm a novice developer, and I go out and look for samples and documentation in order to piece together a simple product list from my database. As long as we do our work right (and we plan to), the documentation will lead me to write something like this:

@{ var products = db.Query("SELECT * FROM Products WHERE CategoryId = @0", categoryId); }

@foreach(var product in products) {
    <li>@product.Name</li>
}

It works, and it's SQL Injection safe. Note that although David didn't blog directly about this, Microsoft.Data fully supports parameterized SQL and it actually supports it better than traditional ADO.Net (note that I don't have to fiddle with SqlParameter objects).

Now, I'm learning more about proper architecture, and I decide to switch to an ORM and use Linq for my queries. A quick one-liner change and I'm all set:

@{ var products = db.Products.Where(p => p.CategoryId == categoryId); }

@foreach(var product in products) {
    <li>@product.Name</li>
}

[NOTE: WebMatrix doesn't include a true ORM, as there are plenty of good options out there, this is just a sample of how one could ramp up from Microsoft.Data to more powerful ORMs]

And if I want to go full-tilt and use ASP.Net MVC, I can move my data access code into a Controller (or even deeper into my architecture) and then a bit of copy-pasting gets me to:

public ActionResult Products(int categoryId) {
    var products = db.Products.Where(p => p.CategoryId == categoryId);
    return View(products);
}

@foreach(var product in Model) {
    <li>@product.Name</li>
}

From there I'm free to keep refactoring things behind further abstractions (Repository patterns, etc.). I've started simple and refactored as necessary to improve the architecture. In every step, I've been able to take a lot of the existing code with me, which traditional ADO.Net and other platforms (like PHP) don't make quite as easy.

The simple fact is this: The audience we're targeting is already using inline SQL, they are perfectly happy to keep doing so and they are not interested in clean abstractions (to the point of finding them complex and unnecessary). Microsoft.Data, and the entire ASP.Net Web Pages framework (the inline page model used in WebMatrix), is an attempt to provide a simple model for web development that provides on-ramps to guide users towards best practices.

Inside Razor - Part 3 - Templates

andrew@andrewnurse.net — Mon, 02 Aug 2010 00:00:00 -0700

One of the features of Razor which hasn't been discussed a lot is the Inline Template feature. Razor includes the ability to provide an inline Razor template as an argument to a method. At the moment, this is only used by the Grid helper in ASP.Net Web Pages (and Scott Guthrie showed it back in his original blog post). However, we don't have much documentation on how to create your own templated helpers yet, so I figured I'd talk a bit about it.

First, let's take a look at what code is generated when we use an inline template. I've written a sample templated helper called "Repeat" which just repeats the content of the template a specified number of times (we'll take a look at the implementation later). The page that uses this helper looks something like this:

<!DOCTYPE html>
<html>
     <head>
         <title>Repeat Helper Demo</title>
     </head>
     <body>
         <p>Repeat Helper</p>
         <ul>
             @Repeat(10, @<li>List Item</li>);
         </ul>
     </body>
</html>

And when we run it, we get the following output:

Let's take a look at the implementation of "Repeat". I wrote it inline in the page in this case, but you could just as easily write it in a static class in App_Code and reference it that way. In Razor, the "@functions" block lets you write code that will be injected as-is into the body of the generated class.

@using System.Text;
@functions {
      public static IHtmlString Repeat(int times, Func<int, object> template) {
           StringBuilder builder = new StringBuilder();
          for(int i = 0; i < times; i++) {
              builder.Append(template(i));
          }
          return new HtmlString(builder.ToString());
      }
}

So, the template is being passed in as a Func<int, object> and when we invoke it, we get back the result of running the template. But, if you look at line 6, you'll notice we're passing in an argument to the template function. Let's take a look at the C# that is generated when we write a call to Repeat. Here's the generated code to match the Razor code in the first code listing:

this.Write(Repeat(10,item => new Microsoft.WebPages.Helpers.HelperResult(__writer => {
     @__writer.Write(" ");
     @__writer.Write("<li>List Item</li>");
})));

It's a little complex looking, but essentially, what's happening is that we're writing out a lambda which accepts a single parameter called "item" (the type of which is determined by the method you're passing it to). When that lambda runs, we construct and return a HelperResult. HelperResult is a class defined in the ASP.Net Web Pages framework, and it's essentially a wrapper around yet another delegate which writes text to a TextWriter. Think of it as a mini Razor template, when you invoke the delegate, it writes the content of the template. The advantage of wrapping the delegate up in the HelperResult class is that we can treat it just like a string in most places since it overrides ToString to return the result of executing the template.

The "item" parameter is used in helpers like the Grid helper to provide the current data item to the template so that it can be used. The Repeat helper passes in the iteration number as this parameter, which we can access from within the template by using "@item"?. For example:

@Repeat(10, @<li>List Item #@item</li>);

Which will render:

In summary, if you want to use Razor templates in your helper methods, just add a parameter with the type Func<?, object> where ? can be any type you want. As an interesting little exercise, try converting the Repeat helper to take an IEnumerable<T> and pass each item of that enumerable to the template, rendering the result.

I've uploaded the Razor file containing my sample helper here: RepeatHelper.cshtml.txt (.56 KB) (Note: To avoid issues with file types on my hoster, it has a .txt extension, just remove that and you're good to go!)

Please feel free to ask questions in the comments, or by emailing me at andrew AT andrewnurse DOT net. I'm also on twitter at @anurse and I keep an eye on the "razor" tag on StackOverflow so there's no shortage of ways to ask!

Using the Razor parser outside of ASP.Net

andrew@andrewnurse.net — Thu, 22 Jul 2010 00:00:00 -0700

When Scott Guthrie originally blogged about Razor, he mentioned that it was fully hostable outside of ASP.Net. The engine itself is not quite as detached from System.Web as we'd like, but it's close and we're going to get it way closer in the next release.

Having said that, you can still host Razor outside of the ASP.Net pipeline with the current beta! It's a little trickier, and you do technically need to reference System.Web. I've written a sample console app that I'm attaching to this post called "rzrc" which takes in a .cshtml or .vbhtml Razor file and runs it through the parser and code generator to produce a .cs or .vb file. I'll walk through the main logic here and go over what each section does.

However, I was not the first to do this! Full credit for that goes to Gustavo Machado, who wrote an excellent post in which he used Reflector to work out how to run the Razor parser and code generator. Well done Gustavo! There are a few things that this version does that Gustavo's doesn't, such as cleaning up the Web-related stuff in the generated code and selecting the language based on the Code Language, but he basically hit it spot on!

The first thing my console app does is get the input file name, extract the extension and look up what Razor Code Language it uses. This is done using the CodeLanguageService class, which is part of the Razor APIs:

CodeLanguageService languageService = CodeLanguageService.GetServiceByExtension(extension);
if (languageService == null) {
    Console.WriteLine("{0} is not a Razor code language", extension);
    return;
}

Then, we fire up the parser and the code generator. A CodeLanguageService is basically a factory for constructing a Code Parser, to parse the code blocks after an "@" and a matching Code Generator to write the final C# or VB class.

InlinePageParser parser = new InlinePageParser(languageService.CreateCodeParser(), new HtmlMarkupParser());
CodeGenerator codeGenerator = 
    languageService.CreateCodeGeneratorParserListener(className,
                                                        rootNamespaceName: "Template", 
                                                        applicationTypeName: "object", 
                                                        inputFileName, 
                                                        baseClass: "System.Object");

When you run the Razor Parser, you must provide it with an object implementing IParserConsumer. This interface has callbacks which the parser will call when it encounters various Razor constructs (more details on the Razor parse tree later). CodeGenerator implements this interface and responds to the these callbacks by generating code. However, it does nothing with the errors, so in the console app, I've written a very simple IParserConsumer called CustomParserConsumer which wraps the code generator and outputs errors to the console. I won't put the code here, but it's in the sample, so take a look there if you're interested.

Now that we've got all the objects we need, we can actually run the parser over the input

CustomParserConsumer consumer = new CustomParserConsumer() { CodeGenerator = codeGenerator };
using (StreamReader reader = new StreamReader(inputFileName)) {
    parser.Parse(reader, consumer);
}

Once Parse returns, the Code Generator will have built a CodeDOM tree representing the generated code during the callbacks, so we know that our code is ready to go. Right now, the Code Generator adds in some web specific things. For example, when we constructed the Code Gneerator above, we gave it an "applicationTypeName" which (in a web context) is the type name of the class defined in Global.asax, if there is one. Since we are trying to generate a template that isn't related to the web, we can get the CodeDOM tree from the Code Generator and remove these things.

codeGenerator.GeneratedCode.Namespaces[0].Types[0].Members.RemoveAt(0);
codeGenerator.GeneratedCode.Namespaces[0].Types[0].BaseTypes.Clear();
codeGenerator.GeneratedCode.Namespaces[0].Imports.Clear();

Finally, we use the CodeDOM to write the code to a C# or VB class file (provider is a CodeDomProvider from System.CodeDom.Compiler):

using (StreamWriter writer = new StreamWriter(outputFile)) {
    provider.GenerateCodeFromCompileUnit(codeGenerator.GeneratedCode, writer, new CodeDom.CodeGeneratorOptions());
}

And we're done! This is definitely more complicated than we'd like, but there are plans to simplify this API significantly in future releases. For the most part, all we've done is left the methods our ASP.Net Build Provider uses open and accessible. I wouldn't bet on these APIs staying around too long, but any API changes from here on should be simplifications. For now though, check out the sample I've attached and play around! Note that you must have WebMatrix installed to use the sample.

I've put some comments in which start with "EXT" which contain tips on how to extend this code to your own use. Please feel free to take this code and use it absolutely anywhere you want! Let me know how your using Razor by either tweeting me at @anurse or email me at andrew AT andrewnurse DOT net.

Download the console app here: rzrc.zip (3.46 KB)

Inside Razor - Part 2 - Expressions

andrew@andrewnurse.net — Mon, 12 Jul 2010 00:00:00 -0700

This is part 2 of my Inside Razor series. Read Part 1 here.

In my previous post, I glossed over one line in my sample:

<li>@p.Name ($@p.Price)</li>

Well, it's finally time to get in to how this is parsed! So, to recap from last time, when we see the "<li>" here, we know that we are parsing a block of markup which ends at the "</li>". The markup parser scans forward until it finds the end tag, but before it reaches it, it sees an "@". So, just as with "@foreach", it switches to the code parser.

This is where things get a bit different. The C# code parser looks at that first identifier: "p" and checks its internal list of C# keywords. Of course, "p" is not a C# keyword, so the C# code parser enters "Implicit Expression" mode. The algorithm for parsing implicit expressions is something like the following:

First, Read an identifier
Is the next character a "(" or "["?
- Yes - Read to the matching ")" or "]", then Go To 2
- No - Continue to 3
Is the next character a "."?
- Yes - Continue to 4
- No - End of Expression
Is the character AFTER the "." a valid start character for a C# identifier?
- Yes - Read the "." and Go To 1
- No - DO NOT Read the ".", and End the Expression

The high-level overview of this algorithm is that an implicit expression is an identifier, followed by any number of method calls ("()"), indexing expressions ("[]") and member access expressions ("."). And, whitespace is not allowed (except for within "()" or "[]"). So for example, these are all valid implicit expressions in Razor:

@p.Name
@p.Name.ToString()
@p.Name.ToString()[6 - 2]
@p.Name.Replace("ASPX", "Razor")[i++]

However, the following are not valid, and the second section (after the arrow, "==>") is the only part that would be considered part of the expression by Razor:

@1 + 1 ==> @
@p++ ==> @p
@p    .   Name ==> @p
@p.Name.Length - 1 ==> @p.Name.Length

This is why we have another syntax for expressions: "@(...)". This syntax allows anything you want within the "()". So, you can write all of the previous examples using that syntax as an escape-hatch:

@(1 + 1) 
@(p++) 
@(p    .   Name) 
@(p.Name.Length - 1)

Once we've identified the expression, we pass it along to our code generator. When generating the code for "@foreach () { ... }", we just dump that code into the generated C# class as-is, but when we identify an expression (either implicit or explicit) we do something a little different. You probably noticed that unlike ASPX, there is only one control construct: "@", there is no "@=" to distinguish code that we run vs. expressions that we render the value of. This is where some of the magic of Razor comes in. If we see "@foreach" for example, we know that "foreach" is a C# keyword, so that block is written as a statement to be executed. When we see "@p.Name" or "@(1 + 1)", we know that they are expressions, so after executing them, we render the result. So basically:

@if, @switch, @try, @foreach, @for, etc. are equivalent to "<% %>"
@p.Name, @(p++), @(1 + 1), etc. are equivalent to "<%: %>"

Another side note is that expressions are equivalent to "<%:" and NOT "<%=". We made a decision in Razor that HTML encoding should be the default, and that if you want to write unencoded strings, you can use the IHtmlString interface that has been blogged about before.

So, with all that background, we can quickly jump back to our initial sample:

<li>@p.Name ($@p.Price)</li>

When we see "@p.Name" we identify that as an expression, but the space before the "(" stops us from interpreting it as a method call. Then " ($" are all markup and when we see the "@", we interpret "@p.Price" as an expression and stop at the ")".

So there's a quick overview of how Razor identifies and parses expressions. In my next post I'm going to discuss hosting the Razor parser outside of ASP.Net. As before, please feel free to leave comments if you have questions, or send me a tweet (@anurse) or an email (andrew AT andrewnurse DOT net).

Quick Update - Microsoft WebMatrix Beta released

andrew@andrewnurse.net — Tue, 06 Jul 2010 00:00:00 -0700

Scott Guthrie just announced the first beta release of Microsoft WebMatrix. I'll leave you to check out his blog post to find out more.

This is also your first chance to try out Razor. We haven't released the MVC View Engine for Razor, but WebMatrix includes ASP.Net Web Pages, a simple page model that uses Razor syntax. After installing WebMatrix, just create a new site, drop a CSHTML file in it, put some code in and go!

I'll post more details later, but for now, check out the post and play with the bits!

Inside Razor - Part 1 - Recursive Ping-Pong

andrew@andrewnurse.net — Mon, 05 Jul 2010 00:00:00 -0700

This is the first of my blog posts about the parser for the new ASP.Net Razor syntax. We've been working on this parser for a while now, and I want to share some of how it works with my readers!

The Razor parser is very different from the existing ASPX parser. In fact, the ASPX parser is implemented almost entirely with Regular Expressions, because it is a very simple language to parse. The Razor parser is actually separated into three components: 1) A Markup parser which has a basic understanding of HTML syntax, 2) A Code parser which has a basic understanding of either C# or VB and 3) A central orchestrator which understands how the two mix together. Note that when I say "basic understanding" I mean basic, we're not talking about full-fledged C# and HTML parsers here. I've joked with people on the team that we should call them "Markup Understander" or "Code Comprehender" instead :).

So the Razor parser has three "actors": The Core Parser, the Markup Parser and the Code Parser. All three work together to parse a Razor document. Now, let's take a Razor file and do a full summary of the parsing procedure using these actors. We'll use the sample that I used last time:

<ul>
    @foreach(var p in Model.Products) {
    <li>@p.Name ($@p.Price)</li>
    }
</ul>

Ok, now we start at the top. The Razor parser is essentially in one of three states at any time during the parsing: Parsing a Markup Document, Parsing a Markup Block or Parsing a Code Block. The first two are handled by the Markup Parser, and the last is handled by the Code Parser. So, when the Core Parser is fired up for the first time, it calls into the Markup Parser and asks it to parse a Markup Document and return the result. Now the parser is in the Markup Document state. In this state, it simply scans forward to the next "@" character, it doesn't care about tags or other HTML concepts, just "@". When it reaches an "@", it makes a decision: "Is this a switch to code, or is it an email address?" This decision is basically done by looking just before and just after the "@" to see if they are valid email characters. This is the default convention, but there are escape sequences to force it to be treated as a switch to code.

In this case, when we see our first "@", it is preceded by whitespace, which is not valid in an email address. So, we now know we are switching to code. The Markup Parser calls into the Code Parser and asks it to parse a Code Block. A Block, in terms of the Razor Parser, is basically a single chunk of Code or Markup with a clear start and end sequence. So, the 'foreach' statement here is an example of a Code Block. It starts at the "f" character and ends at the "}" character. The Code Parser knows enough about C# to know this, so it starts parsing the code. The Code Parser does some very simple tracking of C# statements, so when it gets to the "<li>" it knows it's at the start of a C# statement. "<li>" is not something you can put at the start of a C# statement, so the Code Parser knows that this is the start of nested Markup Block. So, it calls back into the Markup Parser, to have it parse a block of HTML. This creates a sort of recursive ping-pong game between the Code and Markup parsers. We start in Markup, then call into Code, then call into Markup and so on before finally returning back up this whole chain. At the moment, the call stack in the parser looks something like this:

HtmlMarkupParser.ParseDocument()
- CSharpCodeParser.ParseBlock()
  - HtmlMarkupParser.ParseBlock()

(Obviously, I am leaving out a lot of helper methods :)).

This highlights a fundamental difference between ASPX and Razor. In an ASPX file, you can think of Code and Markup as two parallel streams. You write some Markup, then you jump over and write some code, then you jump back and write some Markup, and so on. A Razor file is like a tree. You write some Markup, and then put some Code inside that Markup, then put some Markup inside that Code, and so on.

So, we've just called into the Markup Parser to parse a block of Markup, this block starts at "<li>" and ends at the matching "</li>". Until that matching "</li>", we won't consider the Markup Block finished. So even if you had a "}" somewhere inside the "<li>" it wouldn't terminate the "foreach", because we haven't come far enough up the stack yet.

While parsing the "<li>", the Markup Parser sees more "@" characters, which means even more calls into the Code Parser. And so the call stack grows:

HtmlMarkupParser.ParseDocument()
- CSharpCodeParser.ParseBlock()
  - HtmlMarkupParser.ParseBlock()
    - CSharpCodeParser.ParseBlock()

I'll go into detail on how these blocks are terminated later, because it is a little complicated, but eventually we finish these code blocks and we're back in the "<li>" block. Then, we see "</li>" so we finish that block and pop back up to the "foreach" block. The "}" terminates that block, so we back up to the top of our stack again: the Markup Document. Then we read until the end of the file, not finding anymore "@" characters. And we're done! We've parsed the entire file!

I hope that's made the general structure of the parsing algorithm somewhat more clear. The key take-away here is to avoid thinking of Code and Markup as separate streams and think of them as constructs you nest inside each other. Our next topic will be Implicit Expressions, which is the logic that allows us to detect what parts of "@p.Name ($@p.Price)" are code, and what are markup. I'll give you a hint, we took some inspiration from PowerShell here ;).

Please post any questions or comments in the comments section or email me at "andrew AT andrewnurse DOT net"!

Introducing Razor - A New View Engine for ASP.Net

andrew@andrewnurse.net — Sat, 03 Jul 2010 00:00:00 -0700

UPDATE: Fixed broken examples (I hope :)).

Earlier this morning, Scott Guthrie blogged about a new View Engine we're developing for ASP.Net. As many of my readers know, I joined the ASP.Net team back in October in 2009, and I'm really excited to finally be able to share what I have been working on for the past 8 months. When I joined Microsoft, I was shown some early prototypes for this new syntax and over the course of the next 8 months, we developed it into the Beta we're going to be releasing very soon.

Writing the parser for Razor has essentially been my job for the last 8 months, so I'd like to describe a little about some of the design ideas that went in to it as well as some of the interesting ways we implemented things. This is the first in a few blog posts I'll do about the Razor syntax as well as the Parser design.

Razor syntax is designed around one primary goal: Make code and markup flow together with as little interference from control characters as possible. For example, let's take the following ASPX:

<ul>
    <% foreach(var p in Model.Products) { %>
    <li><%= p.Name %> ($<%= p.Price %>)</li>
    <% } %>
</ul>

Now, let's boil it down to the parts that we actually care about, removing all of the extra ASPX control characters:

<ul>
    foreach(var p in Model.Products) {
    <li>p.Name ($p.Price)</li>
    }
</ul>

Obviously there isn't enough data here to unambiguously determine what's code and what's markup. When we were designing Razor, we started from here and added as little as we could to make it absolutely clear what is code and what is markup. We wanted Razor pages to be Code+Markup, with a little extra stuff as possible. We even used that goal as inspiration for the file extension for C# and VB Razor pages: cshtml and vbhtml.

So, using the C# Razor syntax, the above example becomes:

<ul>
    @foreach(var p in Model.Products) {
    <li>@p.Name ($@p.Price)</li>
    }
</ul>

If you ask me, that's pretty darn close to the previous example. Razor takes advantage of a deep knowledge of C# (or VB) and HTML syntaxes to infer a lot about what you intended to write. Let's take this sample and break it down chunk by chunk to see how Razor parses this document.

<ul>

When Razor starts parsing a document anything goes until we see an "@". So this line just gets classified as Markup and we move on to the next

@foreach(var p in Model.Products) {

Here's where things get interesting. Now, Razor has found an "@". The "@" character is the magic character in Razor. One character, not 5 "<%=%>", and we let the parser figure out the rest. If you skip ahead a bit, you'll notice that there's nothing that indicates the end of the block of code (like the "%>" sequence in ASPX). Rather than having it's own syntax for delimiting code blocks, Razor tries to add as little as possible and just uses the syntax of the underlying language to determine when the code block is finished. In this case, Razor knows that a C# foreach statement is contained within "{" and "}" characters, so when you reach the end of the foreach block, it will go back to markup

<li>@p.Name ($@p.Price)</li>

Now, things get even more interesting. Didn't I just say we were in Code until the ending of the foreach? This looks a lot like Markup, and we're still inside the foreach! This is another case where Razor is using the syntax of the underlying language to infer your intent. We know that after the "{" C# is expecting some kind of statement. But, instead of a statement, we see an HTML tag "<li>", so Razor infers that you intended to switch back to Markup. So we've essentially got a stack of 3 contexts: When we started out we were in Markup, then we saw @foreach so we went to Code, now we've see <li> so were back in Markup. At the closing </li> tag, we know you've finished the inner Markup block, so we go back to the body of the foreach.

Then we see the end of the foreach block, so we go back to the top-level Markup context.

</ul>

And we continue parsing markup until the next "@", or the end of the file. You may have noticed I skipped over a bit in the middle of the <li> tag. I'll save the details of that for my next post, but the essential logic is the same: "@" starts code, and we use C# syntax to tell us when that code block is finished.

It's great to finally be able to share the result of our hard work with everyone. We've worked really hard to try to create a really clean syntax for mixing code and markup and I know I'd love to hear your feedback. Post in the comments on my blog, send me a tweet at "@anurse" or send me email at "andrew AT andrewnurse DOT net".

And I know you're all probably eagerly awaiting a chance to try this out. Don't worry, we'll have a public beta soon that you can try out!

TechEd North America 2010 - Day 1

andrew@andrewnurse.net — Tue, 08 Jun 2010 00:00:00 -0700

Day 1 marked the official start of TechEd North America. And while everyone else was in the keynote, I was scrambling slightly as I realized that our demo machine was completely empty, just a clean install of Windows 7 and Office 2010. Fortunately, Brad Wilson came to the rescue, as he had been given a USB key with Visual Studio 2010, our demos and our rotating PowerPoint deck on it. We snapped it in and got started setting up.

We took advantage of our large-ish screen and internet connection to stream the keynote live, even though it was happening right next door (though given the size of the exhibition hall, right next door is a long way away :)). At the end of the keynote, the great voice in the sky (aka the announcer) told us the expo was open, and in came the swarms of people! Well, kinda. Of course, there were three "foundational sessions" (like mini-keynotes on more specific topics, like Developer Tools, etc.) going on soon after the keynote, and many people had sessions to attend, so it was pretty quiet in that first shift.

At 12:30, when my shift ended, Brad and I and a couple of people he knew from Patterns & Practices wandered over to the Meal Hall (Which is no small feat... the New Orleans Convention Center is 1.5 miles long! Granted, we're only using 1 mile of it :)). There we ate "food" (convention food is a close approximation of food in most cases... it was edible and likely had some nutrients in it... close enough) and then I trekked off to my first (and only) session of the day. Don't get too excited, I wasn't speaking, just attending. No talks for me this time around, but I do plan to get into speaking soon. Also, it is notoriously difficult to get speaking slots at TechEd if you haven't given a TechEd talk before.

I attended a talk on Windows Phone 7, which included a pretty good demo. Aside from the demo it was mostly things I knew about already. By the way, all the talks are going to be posted for FREE viewing by the general public on TechEd Online (http://www.msteched.com/), this talk was "WPH201 - Windows Phone 7: A New Kind of Phone"?. It was only slightly ironic that every once and a while, during the talk, I popped open my iPhone to check the latest news from WWDC :). Still, WP7 looks nice, and I'm basically decided that my next phone will likely not be an Apple phone. It may be a Windows Phone or an Android Phone, but even after the iPhone 4 announcement, I'm not really convinced... yet. I'll save that rant for later though :).

That evening, the exhibition hall was open for a reception at which free food and beverages (read: alcohol) was offered to all attendees, while they wandered around the various booths from Microsoft and third-party Vendors. It was basically all-hands-on-deck for that event, and we were definitely pretty busy. A number of great questions from customers of all knowledge levels, from someone who was just getting back in to Web Development and had no idea what had been going on in ASP.Net, to some folks who used MVC and wanted to know about MVC 2.

After that, I basically just wandered back to my hotel and crashed. It was a long day, and I had another one ahead of me!

TechEd North America 2010 - Days 1 and 0

andrew@andrewnurse.net — Mon, 07 Jun 2010 00:00:00 -0700

I know, it's been many many months since I blogged. I wish I could say it was because I was really busy, still getting settled in Seattle, or because I was working on lots of cool stuff I can't talk about yet. While those are all true (*wink*), the real reason is that I just got a bit lazy with the blogging. Anyway, whirlwind update: I've been working on the ASP.Net team full-time for 8 months now, and I'm working on some cool stuff that I'll be able to talk more about soon.

In the mean time, I can talk about the fact that I'm down in New Orleans right now, working as Staff for the ASP.Net booth at TechEd North America! I'm going to try to blog a bit about the conference, starting with my first three days.

I left on Saturday, two days before the conference actually started (hence "Day -1"?). After my wonderful girlfriend woke up early to drive me to SeaTac airport at 5am for my 7am flight, I was on my way to New Orleans via Houston. The landing in New Orleans was definitely the roughest landing I've ever been in (second roughest: major cross-winds in a Dash-8 on final approach into Ottawa airport). Upon walking off the plane, I felt like I had just walked into a vat of pea soup. Having never really travelled to this part of the continent before, I was unfamiliar with the intense humidly (though I had been well warned).

After a short taxi ride, I reached my hotel, The Hotel New Orleans, conveniently located right next to the Convention Center (thus limiting my time in the un-air-conditioned air). It's a nice hotel, with a cool inner atrium design:

After checking-in, I wandered over to Canal St. and the outskirts of the French Quarter

I made my way back to my hotel and decided to eat in the hotel restaurant. I had been given a list of a few things to try while I was here, one of which was Catfish, so I decided to give it a go. After eating that and washing it down with a glass of Sweet Tea (which is a very descriptive name, and a very tasty drink) I basically just crashed early.

The next day, Sunday, was also pretty relaxed. After a long sleep-in, I wandered over to the Convention Center to register

My only commitment was the staff orientation at 3pm, where we got to check out our booth, located in the Technical Learning Center, which is basically where all the Microsoft teams set up booths.

Past TechEd attendees may remember that in previous events, this was a separate section from the Partner Expo (where all the third-party vendors set up). In 2010, the TLC and the Partner Expo are all in one big Exhibition Hall. After sizing up our booth and meeting the team, I wandered to a nearby restaurant for a Shrimp Po'boy and Hush Puppies (yet more things on my list of foods to try). I also wandered along the New Orleans Riverwalk and snapped a few pics of the Mississippi (which almost form a panorama!)

Then, I crashed again, this time setting an alarm so I could be up in plenty of time for my duties the next day.

Next up: Day 1 - The first actual day of the conference!

A quick update on MaVeriCk

andrew@andrewnurse.net — Thu, 28 May 2009 00:00:00 -0700

I've been busy with some, paid (and therefore higher priority ;) ), work for the past week or so, which has meant I haven't had time to write up my Module Development in Maverick blog post. I'll try to get that out next week. In the mean time, you can pretty much just take the source (available here: http://github.com/anurse/Maverick) and start playing. Creating a new module is as simple as copying one of the existing folders in the "Source/Web/Maverick.Web/Modules" folder, renaming it, and editing the "Module Application" class inside. There's no registration necessary (thanks to MEF).

Have fun, feel free to post questions, and I'll talk to you next week!

MaVeriCk - An Open-Source CMS written in ASP.Net MVC

andrew@andrewnurse.net — Tue, 19 May 2009 00:00:00 -0700

This is a big post, I know, but I think it'll be worth it. If you don't have time to read it all, I organized it by importance, so the important stuff is at the top anyway :).

I've been hinting for a while that I've been working on a secret side project using ASP.Net MVC that was sort-of related to DotNetNuke. Well, it's finally time for me to put some code out there :). Today, I published a source code package containing version 0.1 of MaVeriCk, a Content Management System written in ASP.Net MVC. I'll explain what it is and how to get started in this blog post.

Warning: MaVeriCk is nowhere near finished, and very much an alpha (maybe even a pre-alpha ;). Major features are completely missing (such as user management, and any useful modules, see below for details)

Disclaimer: This project has NO RELATION to the DotNetNuke project. While I am a DNN Core Team member, this is purely a personal side-project. DotNetNuke has inspired some of the things I've done in MaVeriCk, but that's the extent of the connection between the two.

What is MaVeriCk?

Maverick is an open-source Content Management System in the style of *Nuke applications such as DotNetNuke. For the rest of this post, I'm going to assume you're fairly familiar with that style of application. Basically, users create pages and drop little widgets (or "Modules"?) onto those pages to provide features like a Blog, Message Board, or even just a block of HTML.

Where (and how) do I get it?

The project is hosted on GitHub (http://github.com/anurse/MaVeriCk/) so you can feel free to clone the repository and hack away there. Alternatively, if you either don't have Git or don't want to bother with getting the whole repository, you can download version 0.1.0, from the downloads section: http://github.com/anurse/MaVeriCk/downloads. Also on the downloads page are the release notes for this version which include a full installation guide. Unfortunately, there is no automated installation system, but the steps are quite straightforward if you're familiar with ASP.Net.

Feel free to post comments on this blog, or send me email at: andrew AT andrewnurse DOT net

(After all, I'm just a junior programmer just out of college and don't purport to be an expert in the various development patterns and architectures, so if you think something doesn't look right, please let me know!)

How does it work?

The rest of this post will be a, very short, introduction to the Module Framework provided by Maverick. I'll be posting much more detailed information about the guts of Maverick later on, this is just a taste. Feel free to send me email (see above) if you have questions about the rest of the architecture, and don't be afraid to dive into the code!

Let's start by taking a look at a MaVeriCk page:

Here, we see a page with two modules. The first module is an instance of the "Current Time" Module Application, the second is an instance of the "Claim Dumper" Module Application. A Module Application is an MVC Application, with its own Controllers and Views, which handles requests for a particular type of module. If you're familiar with DNN, the "Module Application" is analogous to the "Module Definition" concept in DNN.

Here's a look at the code for the Current Time Module Application:

[Export(typeof(ModuleApplication))]
[ModuleApplication(ApplicationId, ApplicationName, "1.0.0.0", "Displays the Current Time", "Maverick", "~/Modules/CurrentTime/Content/Images/Icon.png")]
public class CurrentTimeModuleApplication : ModuleApplication {
    private const string ApplicationId = "A1FE2A31-0BC9-4B12-9B81-3B75C098EB33";
    private const string ApplicationName = "Current Time";
    protected override string FolderPath {
        get { return "CurrentTime"; }
    }
    protected internal override void Init(MaverickApplication application) {
        base.Init(application);
        RegisterRoutes(Routes);
    }
    private static void RegisterRoutes(RouteCollection routes) {
        routes.RegisterDefaultRoute("Maverick.Web.Modules.CurrentTime.Controllers");
    }
}

Compare this with the Global.asax file with an MVC app, and you'll find it's very similar. The "Init" method is equivalent to "Application_Start" and is called the first time the Module Application is run during the current ASP.Net Application's lifetime. Just as you would in an MVC 'Application_Start' method, the Current Time Application registers routes into a route table for this module application. In this case, the module is using a helper that Maverick provides to register the default route, but if we take a look at that helper, the only difference between the standard MVC route and this route is that we require that the namespace containing the Controllers is specified:

public static void RegisterDefaultRoute(this RouteCollection routes, string routeName, string defaultController, string defaultAction, string defaultId, string[] namespaces) {
    routes.MapRoute(
        routeName,
        "{controller}/{action}/{id}",
        new {controller = defaultController, action = defaultAction, id = defaultId},
        namespaces
        );
}

(Note: there are various overrides to this method, including the one you see used by the CurrentTimeModuleApplication class above)

Routing in Maverick is a little more complex than in standard MVC applications. I'll go into more details on it later, but the gist of it is that a portion of the URL for a request is handled by Maverick, and the rest (if any) goes to one of the modules on the page (called the "Active" or "Selected" module). All the other modules (referred to as "Passive" modules) get an empty URL, and their default route is run.

From there on, a module is surprisingly similar to an MVC application. It has Controllers and Views, just like an MVC app. The big difference is that, in most cases, the ActionResult returned by a Controller does not represent the whole page, just a fragment of it. Maverick collects all the ActionResults for a page and then renders them out one-by-one in the correct location. Of course there are exceptions (for example, if a Module returns a FileResult or a RedirectResult), but that's the core of it.

Missing Features

Maverick is in an early preview state at the moment. There is a powerful authentication model available (or "Identity" model, as I call it), but at the moment, there's only a very simple "Debug" implementation of it (which just automatically logs you in as an Administrator). I have a prototype system which uses the Azure Access Control Service in the code, but it is not active by default (I'll post more about that later).

Maverick does not really include any useful modules, so you're on your own there. Module development is relatively simple, and will be the focus of another blog post. Feel free to check out the other modules for more info. The only major component of Module Development not covered in the provided sample modules is data storage. Maverick does have a system for modules to store data in the Maverick Database, but none of the built in modules do that.

There is a very rudimentary Theming system in place to allow you to create your own templates for the site (similar to DNN Skins).

Despite being touted as a Content Management System, Maverick doesn't actually contain any Content Management features :), no Workflow, no Versioning, no Content Approval. However, those features are planned for the future.

If you have any other suggestions, please let me know!

DotNetNuke Reports Module 5.1 enters the Release Tracker

andrew@andrewnurse.net — Sat, 02 May 2009 00:00:00 -0700

It took almost a week longer than I said it would, but it's finally done. Version 5.1 of the DotNetNuke Reports Module is in the DotNetNuke Project Release Tracker. I'll post some more info over the next week, but for now, here are the release notes:

Reports Module 5.1 for DotNetNuke 5.0.1

Required DotNetNuke Version: 5.0.1 or above

The following issues have been resolved in this release of the Reports Module

RPT-9810: Simple URL Parameter system - A simple URL parameter system has been added which allows users to provide a list of QueryString parameters which will be added to the query. The parameter names are in the following format: @url_[QueryString Parameter Name]. Only the parameters specified in the "Allowed URL Parameters" section will be added.
RPT-5953: XSLT Extension Objects - CLR objects can now be provided to the XSLT Visualizer to use as XSLT Extension Objects. These objects must have a parameter-less constructor and MAY (but are not required to) implement the IXsltExtensionObject interface (found in DotNetNuke.Modules.Reports.dll) in order to receive additional context information.
RPT-7025: Search Indexing fails due to an exception - When there is an error in a Data Source, that exception is also thrown by the Search Indexer. Since Reports Module Exceptions were localized, through the Message property, when the Scheduler caught the exception thrown by the module, it would attempt to localize the text and fail (since there is no HttpContext). This has been corrected so that Data Source errors are no longer masked.
RPT-9543: Exception thrown when opening "Manage Add-Ins" page on DNN 5.0 - The DotNetNuke Extensions Installer API was moved to a different namespace in 5.0. The Reports module has been updated to reflect this change.
RPT-8145: Reports Module does not release file handle - The HTML Visualizer was opening a StreamReader and using ReadToEnd to load the contents, but not closing the StreamReader. It has been corrected to use System.IO.File.ReadAllText. A similar issue in the XSLT Visualizer was discovered and resolved the same way
RPT-7237: Show Info Pane working incorrectly - The "Show Info Pane", "Show Controls", and "Auto Run Report" settings were not being updated if the user is not a Super User, despite the fact that these settings are visible to non-Super Users. This has been corrected in the 5.1 release.
RPT-7236: Critical error when no file name entered for HTML Template Visualizer setting - The HTML Visualizer allowed users to leave the Template file field empty, causing a Critical Error on the view page. The view page has been corrected to avoid causing this error. A similar issue with the XSLT Visualizer has also been corrected
RPT-7238: Show Header field cannot be unchecked - A bug prevented the ShowHeader option from being deselected in the Grid Visualizer.

Also included is an update for the UDT Data Source for the Reports Module, here are the release notes:

Forms and Lists Data Source for DotNetNuke Reports 1.1.0 - Release Notes

This is a minor update to the User Defined Table Data Source for the DotNetNuke Reports Module. It has been renamed to the Forms and List Data Source, to match the new project name. Only one minor change is included: The stored procedure used to retrieve the list of Forms and List modules has been removed and replaced with (safe) dynamic SQL to reduce installation overhead. Requires DotNetNuke Reports 5.0 or above.

Track the progress of the module through the DotNetNuke Release process here: http://www.dotnetnuke.com/Development/ReleaseManagement/ProjectReleaseTracking/tabid/997/ctl/History/mid/3337/ItemID/394/Default.aspx

School is done! On to Reports, and something else ;)

andrew@andrewnurse.net — Tue, 21 Apr 2009 00:00:00 -0700

Well, it's official. Assuming I pass the two courses I haven't received grades from, I completed my final day of continuous education (well, except for the summer breaks/internships) for 18 straight years (K-12 + 5 years of post-secondary) today. I've already got marks for 3 of the 5 courses I took this year, and they were pretty good (couple of A's and a B). This semester has been a bit of a strange one, since I had only one computing science course and it was a "Writing for Computer Scientists" course which involved no programming! (Despite that, it was one of my favorite courses this semester :D). I took introductory courses in Communications, Linguistics and Nutrition as well as a intro to the Physics of Sound and Music entitled "Logarithm and Blues" taught by a two-man team of a musician and a physicist (sorry "Writing for Computing Scientists", this one takes the top slot this semester :)).

Now that I'm finished school, probably for a while (still eying a part-time masters at University of Washington when I've been at Microsoft for a while), I can settle down and finish up a few side-projects and some DNN Reports work. Reports Module version 5.1 is finished from a code perspective and contains a number of bugfixes (details to come) as well as two new features:

XSLT Extension Objects - The XSLT Visualizer now allows you to provide a list of .Net Types which are constructed at runtime and mapped into the XSLT file as XML namespaces.
Experimental and mostly hacky support for QueryString parameters in Report Queries - A number of people have been asking for this, and it's a relatively simple feature in the Reports Module architecture. On the settings page, you can provide a comma-separated list of QueryString parameters which will be passed to the Report Query. The parameters are prefixed with "url_" in the query to distinguish them from the existing parameters. NOTE: Don't forget that many Databases use still another prefix, such as SQL Server ("@"), so for example, the full name of the SQL Server parameter mapped to "foobar" QueryString parameter is "@url_foobar". This is only a stopgap solution to allow for some linking into Reports with parameters, so expect the next version to include a more detailed system (which may include breaking changes which will require that you update your queries). I'll do my best to support an upgrade path, but no promises, so use at your own risk

I just need to do some testing and then I'll put 5.1 into the release process. I expect to do that this week, so depending on how long it takes the diligent reviewers in the release process to make sure that the new version won't obliterate your existing site (and believe me, you don't want to rush that!), it should be out very soon!

Moving on to a new project... I've been hinting to some of the people in the internal project forums at DotNetNuke that I've been working on a top-secret project involving MVC. (This is a pure side-project, not part of the DNN Project at all). I'm very close to releasing something which, I think, is pretty cool. I'll keep the secret for now, but let's say this: I've been working on/with DotNetNuke for a few years, and I've a huge fan of ASP.Net MVC. I've learned a lot from DNN and I think MVC users can benefit from some of that experience ;). I'm hoping to have something ready on this secret project, which I call "MaVeriCk", in the next week or two (but Reports 5.1 is my priority at the moment).

Hope that teased you just enough to stay tuned :P

I'm still here! Just cramming... and working

andrew@andrewnurse.net — Fri, 10 Apr 2009 00:00:00 -0700

Hello avid readers! It's been over a week since I blogged, I know, but I've been cramming for exams. The CMPT 376 assignment I mentioned a while ago has finished, so I'm no longer required to blog for grades. Having said that, after exams, I plan to keep blogging, so keep watching the feed :).

Just to tease you a bit: I've got a cool little ASP.Net MVC side-project well underway that I'm just dying to show off on the blog, so stick around :P

Also, to the Reports Module users out there, 5.1 is very close to being finished. There may be a surprise new feature in there too. Details to follow later :D.

Quantum Computing in .Net "Looflirpa"

andrew@andrewnurse.net — Wed, 01 Apr 2009 00:00:00 -0700

Microsoft is really taking major strides in terms of improving the programming experience for developers on all current and future computing platforms. That's why, earlier today, Microsoft released a beta of the new "System.QuantumEntanglement" library. This library, being considered for integration with .Net 4.1 (which, my sources say, is codenamed "Looflirpa"?), provides features for developers working on quantum computers. I won't go in to too much detail, but this essentially means working in an environment where objects can be in many different states (possibly an infinite number) and where observation may change the state of an object. Microsoft has proposed a few options, but I have some additional ideas.

Microsoft's Proposal

Eilon Lipton, of the ASP.Net MVC team, is proposing a number of new additions to the .Net BCL in 4.1. The first, is a class called "StringOr<T>" which has the following API:

namespace System.QuantumEntanglement {
    public class StringOr<TOther> {
        public StringOr(string stringValue, TOther otherValue);
        public string StringValue { get; }
        public TOther OtherValue { get; }
    }
}

This class is used to encapsulate a value which may be a string, but may be another value. This is a common user-input scenario, since almost all user input arrives as strings, but will usually be converted to another data type. He also proposes a more general class called "SchrodingOr<TDead, TAlive>"?

namespace System.QuantumEntanglement {
    public class SchrodingOr<TDead, TAlive> {
        public SchrodingOr(TDead dead, TAlive alive);
        public TAlive Alive { get; }
        public TDead Dead { get; }
    }
}

This generalizes StringOr<T> to support any two types.

Eilon's not the only one talking about this, Scott Hanselman (well known Microsoft blogger), and Rob Conery (Author of the .Net Object-Relational Mapper: SubSonic) have also posted on this topic. So I figured I'd add my comments to the blogosphere.

StringOr and SchrodingOr are great starts, but what if we need to represent objects which may be in 2 states? What about 3 states? Infinite states? That's where my proposals come in. I propose the following additions to Eilon's library:

Generalizing SchrodingOr

The first is SchrodingOr<T1, T2, T3, T4, T5, ... , Tn> (for infinite n). This allows the developer to represent objects in as many states as they want.

C# 5.0 Compiler Support

I'm also proposing the following language syntax to help developers work with these types. Similar to the way the C# compiler converts "int?" to "Nullable<int>" (which would now be replaced with "SchrodingOr<int, Void>" as it more accurately represents the concept of a type which may or may not have a value), the new syntax takes the following:

int?string?bool?DateTime? foo = GetUserInput(...);

And produces output code which looks like this:

SchrodingOr<int, DateTime, bool, string> foo = GetUserInput(...);

Then, we can defer observation until a later time, using the AsA<T>/AsAn<T> method (since Eilon's Alive and Dead properties no longer work in an infinite state environment). If we just want to observe the type and then make a decision, we can use IsA<T>/IsAn<T> methods (of course, both are required, just in case the type starts with a vowel), which returns a boolean indicating if the object is of the type T. For example:

if(foo.IsA<string>()) { return foo.AsA<string;>; }

Probabilistic Observation

Sometimes, we may want to observe an object only if there is a high probability of it being in the right state, to avoid additional quantum variations. To do this, we can use the CouldBeA<T>/CouldBeA<T> methods. These can, optionally, accept a probability threshold beyond which a object is considered in the right state.

Quantum Snapshots

Unfortunately, in between a call to IsA/An/CouldBeA/An and AsA/An, the state of the object may change. So, we may wish to observe an object and record the state of the object at the time of observation. Since in observing the object we may change its state, our observations must be recorded in "Snapshots"?. The snapshot is an instance of QuantumSnapshotOf<TSchrodingOr> (where TSchrodingOr must be one of the SchrodingOr<T1, ..., Tn> types), and has a property "Taken", which is a DateTimeOffset (after all, we need precise timing here) containing the exact time that the snapshot was taken.

Here's an example combining Probabilistic Observation and Snapshots:

if(foo.CouldBeA<string>()) {
	QuantumSnapshotOf<SchrodingOr<int, string>> snapshot = foo.Observe();
	if(snapshot.Taken.Day == 1 && snapshot.Taken.Month == 4) {
		throw new UnreliableObservationException(snapshot, "The data is unreliable");
	}
	return snapshot.AsA<string>();
}

Summary

I think this is really interesting, forward-thinking, stuff from Microsoft. Eilon has done some fantastic work getting the ball rolling, and its up to us to give our feedback! Microsoft has created a section on MSDN for discussion on these new features: http://msdn.microsoft.com/en-US/QuantumEntanglement/Default.aspx

DotNetNuke on IgniteIT

andrew@andrewnurse.net — Mon, 30 Mar 2009 00:00:00 -0700

Did you know that the largest open-source project on the Microsoft platform was started in Vancouver, Canada? It's true! DotNetNuke is the largest Open-Source community in the .Net by downloads (at least). Since moving to CodePlex (Microsoft's Open-Source hosting platform) in January 2009, DNN has been downloaded over 180,000 times (at time of writing). This makes it second only to a World Of Warcraft Add-On development studio which has been on CodePlex since Sept 2007 (at least that was the earliest check-in I could find). Adding that to the all-time download score on SourceForge (the project's previous host) of 4.3 Million downloads, and you've got the the largest .Net open-source project (and one which I am proud to contribute to)!

And here's the great part for us Canucks, DotNetNuke was created right here (by "here"?, I mean where I live :P) in Vancouver!

Microsoft Canada is running a contest called the "Ignite It" award to honour Canadian developers and IT professionals. DotNetNuke is nominated in the Developer category, and we're looking for your votes! There's only 2 days left, and we're in the top 5 (which will go on to be judged). Still, we'd like to be up at number one before voting ends, so here comes the shameless plug :P. Here's the deal: You can vote once per Windows Live account per day (though there aren't many days left :( ). It only takes a second to vote, and I've even created a TinyURL to share with your friends: http://tinyurl.com/dnnigniteit

Let's push DNN into the top spot!

Windows Live Writer - After 2 Months

andrew@andrewnurse.net — Fri, 27 Mar 2009 00:00:00 -0700

Back in January, I started blogging more frequently due to a course I was taking. As part of that, I started using Windows Live Writer to write all my blog posts. I use the DasBlog engine to power my blog, and it supports one of the many blogging APIs that Windows Live Writer uses (I believe it's MetaWeblog). So, I hooked it up to my blog, and everything just worked!

My favorite feature is the ability to create and save drafts and then publish them on demand. I have inspiration at the strangest times, and often in large batches, so I end up with a couple ideas of things to blog in a day, and no ideas the next day. So, I just create a bunch of drafts, get them ready to publish and then publish them over a longer period of time (like say... a day :P). For example, I wrote this post 5 minutes after the previous post, but I thought it might overload my readers to post it immediately :). So, I save a draft and post it a bit later (when I don't have anything to blog about :D).

If you have a blog, and it supports one of the blogging APIs, and you're running on Windows, you really should try Windows Live Writer!

Virtually "Attending"?? MIX09

andrew@andrewnurse.net — Tue, 24 Mar 2009 00:00:00 -0700

Microsoft's MIX Web Development conference was held in Las Vegas last week and while I would have loved to go, the cost and scheduling just made it impossible :(. Despite that, I've been spending the past few days "attending" MIX virtually! Microsoft posts high quality WMV versions of the talks on the MIX website, perfect for catching up on the sessions. With my Media Center PC hooked up through my Xbox 360, I can browse through my rapidly growing collection of talks and watch them at my leisure. It's almost like being there! A couple of my recommendations are below:

Day One Keynote - I haven't had a chance to watch this yet, but it already has two things going for it. Bill Buxton talking about User Experiences and a live DotNetNuke installation using the Microsoft Web Platform Installer 2.0 Beta.
ASP.Net MVC: America's Next Top Model View Controller Framework - I love ASP.Net MVC, and this is a great intro. Of course, I may be biased since I worked on the team last summer. BTW, version 1.0 was released last Wednesday, and a large chunk of the AJAX stuff is my code (my code is actually in a released Microsoft product! w00t!)
Microsoft ASP.NET Model View Controller (MVC): Ninja on Fire Black Belt Tips - If the previous talk was MVC 101, this is MVC 201 :P. Though I have to be honest, I haven't actually watched this one yet. I'm sure its chock full of useful MVC tricks though!
File|New -> Company: Creating NerdDinner.com with Microsoft ASP.NET Model View Controller (MVC) - I know, yet another MVC talk, but if you're more of a "get to the code" type of programmer, this may be your style. Plus, Scott Hanselman is an excellent presenter.
How'd they do it? Real App. Real Code. Two Weeks. Nothing but .NET - Another Scott Hanselman talk, this one only 20mins long. You hear people talk about "Rapid Application Development" but in this talk Scott shows off a concrete example of a 2 week project that forms part of the Adidas Motorsports portal.
Building Out of Browser Experiences with Microsoft Silverlight 3 - This is another one I haven't actually watched yet but intrigued me. With Silverlight 3, Microsoft is positioning itself to take on both Adobe Flash and AIR.

That's by no means a complete list of great talks, just my first impressions based on the first few I've watched and the abstracts that interested me on the list. Feel free to post your own favourites in the comments!

Final Few Weeks of School!

andrew@andrewnurse.net — Wed, 18 Mar 2009 00:00:00 -0700

I was taking a look at a calendar on Monday and realized something... I only have a few more weeks of school left! I don't graduate until June, but classes finish on April 6th. As a result, my workload has been growing exponentially :P, so I'm putting my side-projects on hold. The only exception is my work on version 5.1 of the DotNetNuke Reports Module, which is coming along quickly and should be finished soon.

I went straight from high-school to university, so its been a non-stop 16 years of school. So, it's going to be a little bit strange when I graduate and actually enter the workforce full-time :). Still, I've got an awesome job to go to, so I'm still pretty excited.

I have to keep blogging for my CMPT 376 (Writing) course, so I'll keep doing that, but I'm putting pretty much everything else (including "Duh," my toy programming language) on hold for a couple of weeks. I'll have a bit more time after classes end :).

Are all CS departments this slow to adopt new tech?

andrew@andrewnurse.net — Wed, 11 Mar 2009 00:00:00 -0700

Having spent 5 years taking courses in the Computing Science department at Simon Fraser University, I've become pretty familiar with the various web applications used to track course projects, submit assignments, etc. Over the same five years, I've seen numerous technological changes, both in the "outside world" and within the University. For example, not too long ago, SFU switched to a central authentication service aptly named "Central Authentication Service" (or CAS). It's a pretty nice service, with a snazzy login screen (below).

It's not only snazzy, but a pretty useful system. I even integrated one of my school projects (an online grade tracking application) with it without any external support. Let's contrast that with the current login screen for "Gradebook"?, an application developed by the CS department and used in most CS courses to track grades

Ok, not as snazzy, but here's the really frustrating thing: My account here is NOT the same as my SFU computing account!

The "Gradebook" system also includes an electronic assignment submission server

(BTW: This is only a small snapshot of the page)

Sure, it isn't snazzy looking, but it's simple and to the point (+10 points). However, it comes with a few caveats: I have to use my "Gradebook" account, not my (so-called) "universal" SFU account (-25 points), and ALL submissions must be in ZIP, GZIP or RAR format (-25 points)! For many of my courses, the electronic submission has been a single file, and the instructor often has to email the students to remind them that they have to wrap their single PDF file (or Word Doc, etc.) in a ZIP file to upload it. I can understand the benefits as far as storage space, and bandwidth consumption, but wouldn't that be better achieved by simply placing a size restriction and allowing any file to be uploaded? Plus, when I do login, I get a page with a drop-down list of courses. Here's the contents:

See that scrollbar? Yeah... I'm not registered in anywhere near that many courses (the one course I'm registered in is a few scrolls away), but it would appear the submission server doesn't know that.

It doesn't end with the student interface. In a course on "Web Information Systems," we were required to develop a web-based application of some sort, and my group chose to build a grade tracking system (partly out of frustration with the current system). Before starting the project, we organized a meeting with our instructor to find out what his frustrations were with the existing "Gradebook" system used by the department. He mentioned that instructors had to rebuild the structure of assignments, exams and their relative contributions to the final grade every time they created a course. He continued to show us a tool he developed which allowed him to create more detailed marks, give comments, and build an email to send off to the student. He had to build a custom tool to perform tasks which I would consider essential to a modern grade tracking system. In a few weeks, we hammered out a "school project quality" system running on Ruby on Rails (ASP.Net MVC wasn't out yet :P) which supported all the features we discussed in our meeting with the prof.

Granted, I know it's not as simple to fix as it is to complain. But, with the tools out there, and the number of open-source and commercial courseware products, I still don't really understand why we're using a system with so many annoying issues...

For the CS students (or recent grads) who read my blog, is my experience atypical? Or are your school services as out of date as these?

Tips and Tricks: Installing Programs to a USB Key

andrew@andrewnurse.net — Wed, 04 Mar 2009 00:00:00 -0800

My laptop is rapidly becoming obsolete, particularly when it comes to disk space :(. I want to install my games on it, so I have something to do ~~in class~~ between classes, but I'm running out space. So, I came up with an idea: Buy a 16GB USB Key, and put a bunch of games on that! I keep my USB keys in my backpack, so they're never too far away. It sounded like the perfect solution, until I noticed that many programs can't be installed on Removable Drives!

So, I came up with a simple workaround, once again using an awesome tool from Sysinternals :). I used Junction to create a junction folder called "C:Program FilesGames" which points to a "Games" folder on my USB Key. The following command line does the trick (assuming F:Games is the folder on the USB key, and it already exists):

junction.exe "C:Program FilesGames" "F:Games"?

Seems to be working so far! I just install the game into that folder, and all the files go to the USB Key. Of course, its not quite the same as a portable app, since it will only run on the machine it was installed on, but it takes some of the burden off of my hard drive. If my laptop supported SDHC, I'd probably use a 16GB SD card and just leave it plugged in to my laptop, but unfortunately, that's not an option.

Quake Live - Fun? Sure. Interesting? Meh.

andrew@andrewnurse.net — Mon, 02 Mar 2009 00:00:00 -0800

There's been a lot of buzz around the Internet about Quake Live. I decided to succumb to the hype and try it out and I must say it is pretty cool. However, a lot of people have heralded this as a unique new way of distributing games, and I have to disagree with that. Quake Live is no different than any other downloadable game, it just happens to use the browser as a host. When you go to http://www.quakelive.com for the first time, you log in and wait in a queue. When you reach the front of the queue, you are prompted to download a plugin (in the form of a Windows Installer MSI file) for your browser. If the queuing thing sounds very FilePlanet-esque to you, it certainly does to me. Then you download the ~~game~~ plugin installer and run it (as an Admin). When it finishes, you have to restart your browser and go back to the website. After waiting in the queue some more, the game loads up and runs in the browser.

How is this process different from installing a regular desktop game? You download the installer, run it as Admin, and then start the game. The only different here is that the game code is running in-process (AFAIK) with the web browser. It really isn't that different! Developing a game like this using an existing plugin like Flash or Silverlight (especially since Silverlight 3 is supposed to include 3D Acceleration support) would be, IMHO, much more innovative.

About the only thing in Quake Live that differs from a regular desktop game is the use of a streaming installer, which downloads a (relatively) small bootstrapper and then downloads the rest as you play a 10minute training level. Even that isn't really unique, as Scott Hanselman points out, Guild Wars has included that feature for a while.

Announcing Reports Module Version 5.1 and "Saltspring"

andrew@andrewnurse.net — Mon, 02 Mar 2009 00:00:00 -0800

I've been pretty busy with school, and unfortunately have not been able to get going on the many new features I want to implement in the Reports Module. However, there are a number of bugs in the current version that prevent it from being used with DotNetNuke 5.0. For this reason, as well as a few other bugs that aren't quite as major, I'm going to be preparing a minor bug-fix release: Version 5.1.

This version will fix the following major issues:

Errors when accessing the Extensions page using the module with DNN 5.0
Many Exceptions are written to the DNN Event Log when the Search Indexer runs over the Reports Module
Duplicate Search Items are added by the Reports Module, causing exceptions to occur when multi-row Reports are indexed

I will also be making minor tweaks and clean-ups. However, no new features will be included in this release. I'm afraid I just don't have the time at the moment (being a full-time student sucks up a lot of time). Please let me know if there are any bugs you feel are urgent and need to be fixed in 5.1 by posting in the Reports Module Forums. (On a side note, I want to remind my readers that the blog comments aren't really the best place to post questions about the Reports Module. I try to keep an eye out for them, but he forums are the best place to do that!)

Despite my time constraints, I have been doing a little thinking about new features for the Reports Module. Since it seems to be the "in" thing now-a-days, I've given the next major update to the Reports Module a codename: "Saltspring" and I've been building a small feature list:

Minor architecture changes to the Extensions (Data Sources and Visualizers) to improve testability
Major architecture overhaul to the main module to improve testability, and thus quality, of the core module
Investigate replacing the ZedGraph Chart component used by the Chart Visualizer with the new Microsoft Chart Control
Introducing a new Parameters framework which allows users to define custom SQL Parameters for their queries, which pull their value from various different sources (DNN Objects, Query String, Text Fields, etc.)
- This framework would be extensible, so that new "Parameter Sources" can be installed, just as Data Sources and Visualizers are

Again, if you have any more ideas, please let me know!

Campus Printing: A Tale of Panic and Frustration

andrew@andrewnurse.net — Fri, 27 Feb 2009 00:00:00 -0800

11:00AM - I look in my backpack to check that the Project Proposal that is due at my PHYS 192 class at 11:30 is still in there

11:01AM - I realize it isn't there, I left it at home

11:02AM - I decide to just reprint it on campus, and relax

11:05AM - I fire up Word and open up the Print Dialog

11:06AM - I discover that the campus print server isn't in my printers list

11:08AM - I locate the campus printing instructions and start setting up the printer

11:10AM - I discover that the printer driver I need isn't included in Windows

11:12AM - I locate the printer driver online and start downloading it

11:15AM - Printer driver installed, I configure the campus printer

11:17AM - Still reasonably calm, I fire up Word again and send my document to the print queue.

11:20AM - After packing up my machine, I walk over to the printing station in the Laptop Lab

11:21AM - I discover a "Out of Order - Sorry for the inconvenience" sign :(

11:25AM - I walk to the other side of the building to another lab and put my printer card in the reader

11:26AM - Spending the last $0.11 on my card, I print my document

11:27AM - Calmly, I grab the printed document and see the following text:

Error: PCL Protocol Error

11:28AM - Scramble to a PC in the computer lab and log in to the machine

11:30AM - Machine finally logs in, and I fire up Internet Explorer, navigating to my Live Mesh Desktop

11:31AM - Open up the document from Live Mesh and print it

11:33AM - Go back to the printer station, and discover that my printer card is empty!

11:34AM - Walk to the nearby printer card top-up machine and see: "Out of Order - Sorry for the inconvenience"

11:35AM - After discovering I had no cash, walk to the nearby ATM and withdraw some

11:37AM - Head to the nearest cafeteria and buy a bottle of water, getting a twoonie ($2 coin, for my non-Canadian readers) in change :)

11:37AM - Head off to the library on the other side of campus (where the only other printer card top-up machine is, AFAIK)

11:40AM - Reach the library and stick my card in the machine, pop my twoonie in the slot and top-up my card

11:41AM - Go over to the library printer and release the printer job

11:42AM - Pick up the paper and dash over to the lecture hall with my printed proposal in hand

11:45AM - Arrive, late, and take the only remaining seat, at the front of the class, right next to the pile of proposals from the other students

11:46AM - Drop my proposal on the pile and sigh in relief

Seriously, we need a way better campus printing system. Only two top-up machines on campus? Why can't I just add funds to my student account online and submit PDFs and DOCs to be printed if I don't want the hassle of configuring the print server?

Arg...

Tips and Tricks: Remotely Enable Remote Desktop

andrew@andrewnurse.net — Mon, 23 Feb 2009 00:00:00 -0800

Before I start this post, I need to say one thing: Mark Russinovich (as well as anyone else who works on the Sysinternals stuff) is a Windows Magician. The stuff these tools can do is just plan magic :).

Last week, my Dad's Dell XPS laptop decided it was tired of rendering graphics and the video card just stopped working (apparently this is a known issue). Dell support was fairly helpful and on Friday we received a box to sent the computer in for a motherboard replacement. Now, I'm paranoid when it comes to data, I don't trust anyone. So, I suggested we try and wipe the hard drive of all the sensitive data on the machine, and change the password so that we wouldn't have to give Dell the actual password. Now, only the graphics card was damaged, the computer ran just fine and even the graphics were intermittent (at first). We tried to get the graphics card working long enough to enable remote desktop, so we could connect from his screaming new Dell desktop (Quad Core i7 with 12GB of RAM... I'm jealous). Unfortunately, by that time, the graphics card was done... no dice :(

After a little searching, I came across some tips for enabling remote desktop... remotely. It turns out, there are a number of ways to connect to a machine and make changes remotely! The first tip I found was to use the remote registry to enable it: http://www.windowsdevcenter.com/pub/a/windows/2004/05/04/serverhacks_remote.html. Unfortunately, the Remote Registry service wasn't running on the laptop! That's when I found Sysinternals "PsService" tool. The tool allows you, as long as you have the Admin account password, to enable/disable and change the start mode of services on a remote machine. So, first I enabled to remote registry service

psservice bilbo -u [UserName] start RemoteRegistry

Then, just for good measure, I set the Remote Desktop service to autostart

psservice bilbo -u [UserName] setconfig TermService auto

Finally, rather than fight with creating a firewall rule remotely, I just disabled the Windows Firewall service (don't do this unless you absolutely have to :D)

psservice bilbo -u [UserName] setconfig MpsSvc disabled

(Btw, I got all these service names using the Services console on another machine)

Finally, I followed the instructions I mentioned earlier to set the right registry flags to allow Remote Desktop. I killed the power to the PC (I actually tried Sysinternals "PsShutdown" tool first, but had less luck getting that working :D) and restarted it. When I heard the Windows start up sound (as I said, the rest of the components worked just fine :), I fired up remote desktop connection and... bam! I had a remote desktop screen. After a quick pass over the data drive with yet another Sysinternals tool: "SDelete" and a password change, we shut the computer down, packed it up and sent it off to Dell, confident that the important data was scrubbed clean.

Like I said... black magic :). But seriously, the Sysinternals tools rock! If you don't have them, get them now!

Compiler Project - Part 4: ASTs and switching to MGrammar

andrew@andrewnurse.net — Wed, 18 Feb 2009 00:00:00 -0800

Here's part 4 of my infinite part series on developing a compiler from scratch. See all the posts here.

I've decided to change things up a bit on the compiler project. Instead of writing the tokenizer and parser from scratch, I'm going to check out a new language that's part of Microsoft's "Oslo"? Project. Oslo includes a language called M, which is primarily designed for developing data models. M is actually three languages, MGraph, MSchema and MGrammar. MGraph is a language for defining abstract graphs, for example:

Universities {
    Canada {
        BritishColumbia {
            SFU { 
                Name { "Simon Fraser University" }, 
                Location { "Burnaby, BC" } 
            },
            UBC { 
                Name { "University of British Columbia" }, 
                Location { "Vancouver, BC" } 
            }
        }
    }
}

MSchema is a language for defining the schema for these graphs, for example

module Universities {
    type University {
        Name : Text#100,
        Location : Text#100
    }
}

Finally, MGrammar is a language for defining custom Domain-Specific Languages (DSLs) which map to MGraph graphs. This is the language that is most useful to this project. After all, a general-purpose programming language is just a DSL where the "Domain" is very broad :). So, I've spent a few hours this week whipping up an MGrammar grammar for Duh, and the results are promising. I'll be posting some details on that in the next post, but first we need to explore a key concept in compilers, Abstract Syntax Trees.

The end goal of the tokenization and parsing phases of a compiler is to produce an Abstract Syntax Tree (or AST) representing the source code. This is probably best illustrated by an example, so consider this expression:

5+6

This is an addition operation, using 5 and 6 as the arguments, which we represent in an AST like this (using the MGraph syntax above):

Add { 5, 6 }

Similarly, more complex expressions can be translated to trees:

5+6*(3+4)

Becomes:

Add { 5, Multiply { 6, Add { 3, 4 } } }

Which represents the following instructions:

Add 3 and 4

Multiply that result by 6

Add 5 to that result

Essentially, we are building up the expression from the bottom of the tree up. This makes it much easier to convert the expression into code.

I'll post the MGrammar file for the language when I finish it up in a day or so, then its on to writing the code to do the parsing.

Get-PowerKit: A PowerShell Packaging System

andrew@andrewnurse.net — Mon, 09 Feb 2009 00:00:00 -0800

UPDATE 2: PowerKits requires PowerShell 2.0. If you're seeing errors regarding the "try-catch" keywords, its because those keywords were added in PowerShell 2.0 and I use them in PowerKits. Sorry!

UPDATE: It turns out PowerShell 2.0 already contains a feature like this: "Modules."? See here for more info. Since it seems to be far superior to PowerKit (which I just hacked together for my own needs), I guess that just leaves this as an interesting exercise :).

I've been a huge fan of Windows PowerShell since it first came out as "Monad" (aka "MSH"?). Now that its entering version 2.0, which will be built-in to Windows 7 and Windows Server 2008, I'm getting even more into it. I've written a Podcast Receiver (PowerCast) and an automated process for converting videos for use on my Xbox 360 (PowerCode).

[The following paragraph is a short intro to PowerShell, feel free to skip it if you're familiar with PoSH]

What is PowerShell? PowerShell is an object-oriented shell which, IMHO, kicks the pants off of BASH and Perl. PowerShell is a dynamically-typed scripting language written on the .Net Framework, which gives you full access to every class, method, and property in the framework. You could write a GUI, Server, heck even a 3D Game in PowerShell (it just might not perform as well as another language). PowerShell is also an interactive "Read-Eval-Print" loop application which gives you a prompt into which you type commands which are then evaluated and their results printed to the screen (i.e. a standard "Shell" similar to cmd.exe or BASH). Finally, PowerShell is an embeddable scripting environment which lets any .Net-based application embed the PowerShell scripting language. However, most people (myself included) are referring to the Shell itself when they talk about PowerShell

Anyway, on to the main purpose of this post. I started to get frustrated with trying to manage all my scripts and keep them organized. I was also downloading a lot of useful scripts off of the web, but had to manually place them somewhere in my file-system (let alone managing all the dependencies). Other scripting environments, such as Ruby, have packaging systems, such as RubyGems, to manage libraries of functionality and publish them for others to download. So, I thought, why doesn't PowerShell have something like this?

Thus, PowerKits were born. The concept is simple. A PowerKit is a directory on your file-system which contains a manifest file "Kit.xml"?. This file specifies some information such as the paths to various script files, aliases to create, .Net assemblies to load and (in a future version) PowerShell "snap-ins" to load. After installing a kit (by placing it in one of many, configurable, pre-defined folders), it can simply be enabled using a single PowerShell Command. For example, to enable the "MyCompany.MyKit" PowerKit:

Enable-Kit MyCompany.MyKit

Enabling a kit loads all the assemblies, script files, etc. into your PowerShell environment, making them available for use. A Kit can then be disabled similarly using the "?Disable-Kit" command. Disabling the kit removes all the script files, etc. from the environment (NOTE: .Net assemblies cannot be unloaded without restarting PowerShell, due to a limitation of the .Net Framework).

PowerKit also supports packaging kits into a ZIP-like format, specifically the Open Packaging Convention developed by Microsoft for Office 2007 (since the .Net framework has built-in support for that format). These "Kit Archives" can be downloaded and installed using a single command. Eventually, I plan to add support for installing a kit directly from the web, so you could use a command like this:

Install-Kit http://www.mycompany.com/MyCompany.MyKit.kit

Or even this:

Install-Kit MyCompany.MyKit

And the framework would go out and download the kit and install it, giving PowerShell a true "gems-esque" packaging system.

To install PowerKit, download the Install-PowerKit.ps1 script from the CodePlex site, more installation details are in the release notes on that page.

Argument check gotcha with "yield" keyword in C#

andrew@andrewnurse.net — Mon, 02 Feb 2009 00:00:00 -0800

I recently used the 'yield' keyword in a C# method and ran into a strange issue when trying to verify arguments. I always try to verify the arguments provided to my methods so that I can throw ArgumentException or ArgumentNullException as necessary. However, when you use the 'yield' keyword to create an enumerator in C#, you can run into issues

The 'yield' keyword lets you easily create a method which returns an enumerator (which can then be used in a foreach loop). For example, if I wanted to make a method that took a DirectoryInfo (representing a directory on disk) and returned file names of all the .txt files in it, I would write this:

public IEnumerable<string> GetTextFiles(DirectoryInfo dir) {
    if (dir == null) {
        throw new ArgumentNullException("dir");
    }
    foreach (FileInfo file in dir.GetFiles("*.txt")) {
        yield return file.Name;
    }
}

However, the C# compiler will do something tricky with this method, because of the 'yield' keyword. It will create a custom implementation of IEnumerable<string> class that uses "magic" (I'm not going to go into detail as to how it works :P) to jump in and out of the code you wrote. Then, it compiles our method to the following (as decompiled with RedGate's .Net Reflector, a free tool every .Net developer must have :D):

public IEnumerable<string> GetTextFiles(DirectoryInfo dir)
{
    <GetTextFiles>d__0 d__ = new <GetTextFiles<d__0(-2);
    d__.<>4__this = this;
    d__.<>3__dir = dir;
    return d__;
}

Wait a sec, where'd our code go? It looks like it's gone into this <GetTextFiles>d__0 class. If you think that name sounds compiler-generated, you're right! It's the custom implementation of IEnumerable<string> that was created by the compiler. Cool, eh? However, there's a side effect here. Where did our argument checking code go? Unless there's something in the <GetTextFiles>d__0 constructor, the user could pass null in for the dir parameter and no exception would be thrown! So, let's take a look at the <GetTextFiles>d__0 constructor (using Reflector again):

[DebuggerHidden]
public <GetTextFiles>d__0(int <>1__state)
{
    this.<>1__state = <>1__state;
    this.<>l__initialThreadId = Thread.CurrentThread.ManagedThreadId;
}

Nope, nothing there. If you keep checking around the <GetTextFiles>d__0 class, you'll find it in the MoveNext method on that class. That means that instead of checking the argument when you call the method, the argument won't be checked until the first time MoveNext is called (i.e. the first iteration of a foreach loop). If, instead, we wrap the actual "yield" code in it's own method, and call it from our original method (after argument checks), we'll get the desired result:

public IEnumerable<string> GetTextFiles(DirectoryInfo dir) {
    if (dir == null) {
        throw new ArgumentNullException("dir");
    }
    return InternalGetTextFiles(dir);
}
private IEnumerable<string> InternalGetTextFiles(DirectoryInfor dir) {
    foreach (FileInfo file in dir.GetFiles("*.txt")) {
        yield return file.Name;
    }
}

Now, the "magic" is occurring in InternalGetTextFiles and GetTextFiles is compiled without modification and our argument checking works just fine.

Wild Ideas - Using lambdas to check arguments in C#

andrew@andrewnurse.net — Fri, 30 Jan 2009 00:00:00 -0800

One of the most common checks I do is a simple null check on arguments being passed in to the methods I write. I usually create a static class called "Arg" with the following methods (to help out):

public static class Arg {
    public static void NotNull(object value, string paramName) {
        if (value == null) {
            throw new ArgumentNullException(paramName);
        }
    }
    public static void NotNullOrEmpty(string value, string paramName) {
        if (String.IsNullOrEmpty(value)) {
            // ED: Error_StringArgumentNullOrEmpty is a key in my Visual Studio
            //  project's default string resources file (Properties/Resources.resx)
            throw new ArgumentException(
                String.Format(CultureInfo.CurrentCulture, 
                              Resources.Error_StringArgumentNullOrEmpty, 
                              paramName), 
                paramName);
        }
    }
}

Then, I can use it like this

public void Foo(string arg, object reallyLongArgumentName) {
    Arg.NotNullOrEmpty(arg, "arg");
    Arg.NotNull(reallyLongArgumentName, "reallyLongArgumentName");
}

I was recently playing with lambda expressions in C# and thought of a way to make this a little cooler. The result is, I can rewrite the lines above like this:

public void Foo(string arg, object reallyLongArgumentName) {
    Arg.NotNullOrEmpty(() => arg);
    Arg.NotNull(() => reallyLongArgumentName);
}

The first obvious advantage is that for longer argument names, its more concise. The second, is that I no longer have to worry about updating the strings representing the parameter names when I rename arguments, the Visual Studio Refactoring engine will take care of it for me.

Obviously, this is slower than the previous method since I'm diving into the lambda expression at runtime, but its cool. But if I need the performance, I have a little Regular Expression I can run in Visual Studio to convert the lambda calls back to the string versions (and vice-versa)

How does it work? Well, the core of the code is a static helper in the Arg class

private static string GetParameterName<T>(Expression<Func<T>> paramExpr) {
    LambdaExpression lambda = paramExpr as LambdaExpression;
    Debug.Assert(lambda != null);
    MemberExpression paramRef = lambda.Body as MemberExpression;
    Debug.Assert(paramRef != null);
    // Get the parameter name
    string paramName = paramRef.Member.Name;
    return paramName;
}

Lines 2-5 dive through the Expression tree to find the MemberExpression that represents the parameter (i.e. "foo" in () => foo). Then, we pull out the MemberInfo for the parameter and check the name. With that method, the actual checker is easy:

public static void NotNull(Expression<Func<object>> paramExpr) {
    string paramName = GetParameterName(paramExpr);
    // Compile the lambda (to get the value)
    Func<object> compiledLambda = paramExpr.Compile();
    
    // Run the contract check
    NotNull(compiledLambda(), paramName);
}

Line 2 extracts the parameter name. Line 5 compiles the lambda into an actual function that will return the value of the parameter. Finally, Line 8 uses the value and the parameter name to call my original object/string version. The code for NotNullOrEmpty is nearly identical.

Anyway, if you're concerned about performance, stick to the overloads which take the parameter name directly. I've attached the code as a TXT file, just rename to C#, change the namespace as appropriate and enjoy

Arg.txt (3.93 KB)

Compiler from Scratch - Part 3: Tokenization Overview

andrew@andrewnurse.net — Wed, 28 Jan 2009 00:00:00 -0800

Here comes part 3 of my infinite part series on developing a compiler from scratch. See all the posts here.

The first phase of compilation is called Tokenization. Tokenization is the process of taking individual characters in the source code file and grouping them into more conceptual "Tokens." We're not parsing the language, or checking syntax, we're just grouping related characters together. Why do we do this? If we skipped this step, we'd be passing characters directly to our parser. The parser doesn't care that the text contains the characters 'f' 'o' 'o', it cares that the source file contains an Identifier called "foo". There's also a separation of concerns issue. In theory, if you wanted to allow accented Unicode characters like ?, ?, ?, etc. (you may see blocks or '?' characters instead of the real characters, if so just pretend they're there :D) as valid characters in identifiers, you should only have to change the Tokenizer (to accept those characters and use them, along with the regular alphanumeric characters, to build identifier tokens). The formal name for Tokenization is actually "Lexical Analysis," and sometimes the Tokens are called "Atoms."

So, how do we Tokenize text? There are lots of tools out there, most of which let you define your tokens using Regular Expressions (so an identifier would be: "[_A-Za-z][_0-9A-Za-z]*", one letter or underscore followed by zero or more letters, numbers or underscores). These tools build state machines (aka "Finite Automata") that basically walk through all the Regular Expressions until exactly one of them matches the current text buffer. Tools like "lex" basically allow you to build a full-featured, efficient, Tokenizer by simply writing Regular Expressions.

However, that would be too easy for this series :). Instead, I'm using a very simple, hand-coded, Tokenizer which walks through the characters, using "if" and "switch" statements to decide what tokens to output. However, we can't just simply iterate through the characters. Consider the "->" operator in Duh (see Part 1). Duh supports the simple arithmetic operators, specifically "-", and I plan to support inequality operators (i.e "<", ">", etc.), so in theory, the "->" operator could be outputted as a pair of tokens: MINUS and GREATERTHAN (I'll use ALLCAPS to denote the names of tokens). However, this would be putting a larger burden on the parser to detect the MINUS GREATERTHAN pattern. Instead, we can take advantage of "lookahead"

The Tokenizer uses the .Net TextReader class, which lets us walk, one character at a time, through the characters in a body of text. However, once we've called the Read method, the TextReader moves on to the next character and the next time we call it, we'll get the next character. So, in order to properly parse the "->" operator, we need a way to look at the next character, without moving the reader. Fortunately, the TextReader also has the Peek method, which does exactly that.

So now, when we encounter a "-" symbol, we look at the next character. If it is a ">", we output an ARROW token, and move to the next character. Otherwise, we output a MINUS token and leave the reader at the ">" character. Then, in the next iteration, we read the ">" and output a GREATERTHAN token.

Phew, that was a bit long, and more theoretical than I like, but I'll post some code in the next part. Next time, we'll look at the actual code for the Tokenizer.

ASP.Net MVC 1.0 Release Candidate released

andrew@andrewnurse.net — Wed, 28 Jan 2009 00:00:00 -0800

Phil Haack (program manager on the ASP.Net MVC team) just posted a blog post about the new Release Candidate of the MVC framework. Since I got a chance to work on the MVC framework team as an intern this summer (and I'm going back full-time this summer), I'm pretty excited about the news. When this goes RTM, it'll be the first time code that I wrote will be part of a full shipping Microsoft product!

So download the binaries from ASP.Net or check out the source code on CodePlex and have some fun!

Also, check out Scott Guthrie's post on the new features. He shows off one of the features that, I think, makes ASP.Net MVC more compelling than Ruby on Rails: Powerful first-party tooling support.

Compiler from Scratch - Part 2: Normalizing Newlines

andrew@andrewnurse.net — Sun, 25 Jan 2009 00:00:00 -0800

Well, this is a good sign, I'm posting part 2 :). As I said in Part 1, there are no promises here. This series may not go anywhere, I may have to drop it due to my other commitments (school, work, etc.).

In the next few posts, I'm going to talk about the Tokenization phase of the compiler. Before I go into too much detail on that though, I want to talk about newlines.

In Windows, a new line in a text file is indicated by a pair of characters: A Carriage Return (commonly referred to as "\r", as that is the C/C++ string escape sequence for it) followed by a Line Feed ("\?n"?). However, in Unix-based operating systems, a new line is often indicated by a Line Feed character alone. To add even more confusion, the Mac OS uses a Carriage Return character alone.

A compiler needs to track line numbers accurately, in order to report errors, so we need be extra careful around newlines. We could simply use the current operating system's default newline characters, but that makes it difficult for multi-platform development. Instead, we'll normalize the newlines so that all three different types are properly understood by our compiler.

To do this, I've written a "Decorator" class which inherits from the abstract TextReader class provided in the .Net framework. This "Newline Normalizing" decorator wraps an existing TextReader and does all the work of normalizing new line characters TextReader provides two methods that need to be implemented, Read and Peek. Read returns the current character from the text and moves the reader one character forwards (so that the next call to Read will return the next character). Peek also returns the current character from the text but does not move the reader forward. The "Newline Normalizing" reader implements these two methods using the following code

public override int Peek() {
    // Get the next character
    int i = Adaptee.Peek();
    
    // If the character is a 'r' newline, just return 'n'.  
    // Unlike Read, we aren't going to read ahead to check for rn
    // because that will happen when the user calls Read()
    if (i == (int)'r') {
        i =  (int)'n';
    }
    return i;
}
public override int Read() {
    // Get the next character
    int i = Adaptee.Read();
    // If the character is a 'r' newline, we're going to normalize it to 'n'
    // However, if the newline is 'rn', we need to return it as one character, so
    // we check ahead for that
    if (i == (int)'r') {
        if (Adaptee.Peek() == (int)'n') {
            Adaptee.Read(); // Skip the 'n'
        }
        i = (int)'n';
    }
    return i;
}

Essentially, if the character we read from the source (the "Adaptee" as I call it) is a '\n', we just pass it along. If the character is a '\r', we are going to return '\n', but first we first check to see if it is immediately followed by a '\n'. If it is, we skip the extra character. The result is that no matter which newline sequence is used, this TextReader will return it as a single '\n' character.

Next post, I'll start talking about the Tokenization process.

Writing a Compiler from Scratch - Part 1

andrew@andrewnurse.net — Mon, 19 Jan 2009 00:00:00 -0800

EDIT: Whoops, wrote the wrong course number (CMPT 376), corrected below. I'm taking CMPT 376, Writing for Computer Scientists, this semester, must have gotten myself mixed up :)

Ok, I'm going to try something here, and I don't know if I'll manage to finish it. My all-time favorite class in my school career (so far) has been CMPT 379 - Compiler Design. The class basically consisted of me writing a compiler in Visual C++, with weekly lectures and code reviews from the professor (it was a small class). Since then, I've been trying to think of a reason to write a compiler (i.e. a language that can be ported to .Net, or a useful domain-specific language, etc.). Then, this morning, as I walked back to the Computing Science common room from the Gym (after my morning workout), I found a reason. I think compilers are some of the most interesting pieces of software, and I thought that if I wrote a series of blog posts in which I developed a compiler, maybe I could share that passion with others.

So, here's the plan: I've designed an, extremely simple, language that I'm going to walk through writing a compiler for. The initial version will target .Net IL code (since it's a stack machine system, which is much easier to generate code for). Once I've implemented the simple language, I think it'd be awesome if my readers could jump in and propose some ideas for new language features, and I'll try to blog about implementing them.

To be honest, this project may well fall flat on its face. Writing a compiler is not a simple task, no matter how simple the language. However, I do think it will be fun (while it lasts), so let's give it a try!

So, step one is: Define your language! The most essential part of compiler design is having a very clear idea of the purpose of your language. The language I'm going to design is called "Duh"?, because it's probably about as simple as it gets (maybe a little more complex than Brainf**k (NOTE: link target does include a few four-letter words :D)). Here's a sample Duh program

print "Enter your age: ";
ageInput : string;
age : int;
ageInput = readline;
age = ageInput -> int;
print "In 10 years you will be " + (age + 10) -> string + ". Wow, that's old"

Pretty simple, eh? Line 1 is a simple print statement, which accepts a string and write it to the console. Lines 2 and 3 declare variables of string and 32-bit integer types (decided to use a more Pascal-esque variable declaration format, just for fun). Lines 4 and 5 assign values to those variables. Line 4 uses another built-in function, readline, which reads a line of text from the console. Note that we don't even support initializing variables in the declaration! That's ok though, this is just a toy language, we can add initializations later. Line 5 introduces the "->" conversion operator, which takes the value on the left and converts it to the type on the right. It's not quite the same as a cast, because it will try to convert the value, if it can. Finally, we add 10 to the value the user entered and convert it to a string, then we place that string into a large message and print it to the console.

Well, here goes nothing! Feel free to post your initial comments. I hope you'll follow along! I'll be posting full source code with every post

My First Windows 7 Blue-Screen - But it was my fault :P

andrew@andrewnurse.net — Thu, 15 Jan 2009 00:00:00 -0800

I just had my first Windows 7 Blue Screen of Death :(. However, it was my own fault for trying to use an unsupported driver :P. I had just listened to an episode of Security Now on a tool called Sandboxie. Sandboxie is a tool which intercepts Windows API calls made by programs to read/write files and registry keys. Once intercepted, Sandboxie redirects them to files and registry keys in a special "Sandbox"?, so that changes made by programs are isolated from each other. It's a cool security solution, and very similar in spirit to Microsoft's App-V platform (only cheaper, and aimed at consumers rather than enterprises :D).

In order to intercept the Windows API calls, however, Sandboxie has to install a kernel-mode driver and patch the kernel. In 64-bit versions of Windows, a system called PatchGuard prevents this from happening, thus Sandboxie is not compatible with those operating systems. However, my laptop is running a 32-bit version of Windows 7, so I decided to try it out.

At first, I got a compatibility message from the Sandboxie installer, telling me that my OS is not supported. That should have been my first clue :). I decided to take a gamble and try it out anyway, so I tweaked the compatibility settings for the installer so that it ran in "Windows Vista" compatibility mode. The installer ran fine, and installed the software. However, when I tried to run it, BAM, BSOD :(.

Resigning myself to the fact that it just wasn't ready for Windows 7, I booted up in Safe Mode. However, I was unable to run the installer again to remove it. I tried "Add/Remove Programs" and running the installer I downloaded again (in Vista compatibility mode). Still nothing. Fortunately, I was just about to restart for Windows Update when I installed Sandboxie, and Windows Update automatically creates a System Restore point before installing updates. I fired up System Restore, picked the Windows Update restore point and let it do its thing. The machine rebooted, and I was back in action, with Sandboxie (and my BSODs) gone. I had lost the updates that WU installed, but that's a minor inconvenience.

Anyway, all is well now, and I was able to boot up again (in order to write this post in fact :D). So, two lessons here:

Use System Restore! Just remember to make restore points before installing software that you are concerned about. (Though that will NOT protect you from malicious software, just incompatible software)
Check out Sandboxie, just not on Windows 7 :(. My theory is that the PatchGuard technology from the 64-bit OS may have been brought into the 32-bit OS.

Windows 7 First Impressions

andrew@andrewnurse.net — Tue, 13 Jan 2009 00:00:00 -0800

So, as an MSDN subscriber (no, I'm not made of money, Microsoft Interns get a free year-long subscription to MSDN for personal use :P), I had access to the Windows 7 public beta a day early. I decided to go crazy, since I've been hearing its really stable, and put the latest OS on both my laptop and my desktop. (Well, I actually put Windows 7 Server, aka Windows Server 2008 R2 on my desktop). So, I figured I'd post my first impressions.

Installation

There's not much to say here, Installation is exactly like Windows Vista, only a little faster. The only new feature is that Windows 7 Setup prompts you to create a HomeGroup, if you want. HomeGroups are the new networking construct introduced in Windows 7 designed to make it easier to share files and devices between networked computers. I haven't had a change to check that out yet, so I'll come back to it later.

Initial Impression

Besides a stylish new boot screen, in which four coloured dots dance around before combining to form the Windows logo, the boot process is also identical to Windows Vista. I did find that it booted up much faster than Vista (though I can't make an accurate comparison, since my laptop was getting a bit overloaded). The new taskbar is very cool, and while it is a bit of a knock-off of the OSX Dock, I think Microsoft has (in typical Microsoft fashion) gone above and beyond the OSX experience. For example, by hovering the mouse over an icon, a list of all the windows belonging to that application appears. Even better, applications which directly support Windows 7 can add their own "windows" to this list. For example, even though I only have one IE8 window open, each tab in that window appears as a separate item in the windows list.

By hovering over each thumbnail, that window is brought to focus on the screen, and the rest of the windows become "glass"?.

(And yes, I did blank out my Windows Messenger buddies list :P).

Jump lists are another cool feature, but I haven't had a chance to explore it much. Essentially, when you right click, or click and drag up on one of these taskbar icons, a jump list appears.

In this case (Internet Explorer 8), my history is displayed. Applications designed for Windows 7, get a lot of control over this list, but applications which are not designed to support it (PowerShell 2.0 for example) just get a simple default list

I haven't had much of a chance to explore the rest of the new stuff, so I'll post more later, but my initial impression is that Windows 7 is just plain awesome :).

Redirecting your Zune Library to another folder/drive

andrew@andrewnurse.net — Sun, 11 Jan 2009 00:00:00 -0800

As I mentioned in an earlier post about backing up a Zune Library (Backup a Zune Library...), I wanted to find a way to be able to actually store the Zune library database on a different drive. I played with the Microsoft Sysinternals "Junction" tool, which lets you create hardlinks (Wikipedia on Hardlinks) and managed to get it to work. Here's the process to do it yourself. Replace UserName with your Windows user name and D:MusicZuneLibrary with the path to the folder you want to store the database in.

Download Junction.exe from here: http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx (replace D:UtilsSysinternalsjunction.exe below with the path you downloaded junction.exe to)
Shut down the Zune software if it is running.
Rename the folder C:\Users\UserName\AppData\Local\Microsoft\Zune to Zune_backup (just to be safe). You may encounter a file locking issue, but I found that if I waited a few minutes it worked.
Copy the Zune_backup folder to D:MusicZune and rename it to Library

Run the following command:

D:UtilsSysinternalsjunction.exe Zune D:MusicZuneLibrary

That should do it! Start Zune up again and your library should be up and running. Now you should be able to reinstall the OS without having to back up the Zune library. I might try to whip up a PowerShell or Batch file to do this, but don't get too hopeful :).

Mappr: Projecting Geographical Points on the Screen

andrew@andrewnurse.net — Fri, 09 Jan 2009 00:00:00 -0800

One of the courses I took last semester (Fall 2008), was "Software Engineering II". In this class, we were required to work in groups to implement a project that the professor specified. We had to go through the whole process, design, implementation, testing (though we could choose any software process model we wanted: Waterfall, XP, Agile, etc.). Our group's project was an application called "Mappr" that would allow users to browse a map. Well, it was a little more than that, but that's all the background required by this post, I'll post more background in future posts.

One of the necessary components in mapping software is called a "Projection". The Earth is round, and Latitude and Longitude co-ordinates are spherical measurements representing points on the Earth. In order to convert those co-ordinates to (x, y) co-ordinates for displaying on a (flat) computer screen, you must project the geographical co-ordinates into screen co-ordinates. One well-known technique for doing this is called the Mercator Projection.

A quick aside: The Mercator Projection is widely known (in geography circles) for being highly inaccurate. However, it is the projection used by most road maps, atlases, etc., both physical and digital.

In Mappr, projection is handled by a component called a "Projection Strategy"?. A Projection Strategy is a C# class (Mappr was written in C#) with two methods: GeoToScreen and ScreenToGeo. Here are the signatures of those methods:

public interface IProjectionStrategy {
    Point GeoToScreen(Point geographicalPoint, int zoomLevel, int tileSize);
    Point ScreenToGeo(Point screenPoint, int zoomLevel, int tileSize);
}

The purpose of each method is straight forward: To take in either Geographical (Latitude, Longitude) co-ordinates or Screen (X, Y) co-ordinates, and convert them to the other. In order to do this, we must know the Zoom Level, which is an integer N indicating that there are 2^N tiles on the screen. We also need the size, in pixels, of each map tile image. This means that the size of the map, in pixels, is given by: 2^zoomLevel * tileSize.

The code to project a geographical point on to the screen is shown below:

public Point GeoToScreen(Point geographicalPoint, int zoomLevel, int tileSize) {
    // Convert to normalized mercator
    double lon = geographicalPoint.X;
    double lat = geographicalPoint.Y;
    if (lon > 180) {
        lon -= 360;
    }
    lon /= 360;
    lon += 0.5;
    lat = 0.5 - ((Math.Log(Math.Tan((Math.PI / 4) + 
                 ((0.5 * Math.PI * lat) / 180))) / Math.PI) / 2.0);
    double scale = (1 << zoomLevel) * tileSize;
    return new Point(lon * scale, lat * scale);
}

This code first normalizes the longitude (X direction) so that it is in the range 0.0 to 1.0 (where 0.0 is the left of the map and 1.0 is the right). Then it does what I like to call "mathy stuff" (the calculations are taken from similar code written in Java) with the latitude to put it in the same range (0.0 is the top, 1.0 is the bottom). Finally, we calculate the scale of the map (height/width in pixels, since the map is technically a square) and then we can use the normalized longitude and latitude as ratios of that scale.

The ScreenToGeo method is similar, the code is below. I won't describe this, but just provide it for reference.

public Point ScreenToGeo(Point screenPoint, int zoomLevel, int tileSize) {
    int pixelSpan = (1 << zoomLevel) * tileSize;
    double lngWidth = 360.0 / pixelSpan; // width in degrees longitude
    double lng = -180 + (screenPoint.X * lngWidth); // left edge in degrees longitude
    double latHeightMerc = 1.0 / pixelSpan; // height in "normalized" mercator 0,0 top left
    double latMerc = screenPoint.Y * latHeightMerc; // top edge in "normalized" mercator 0,0 top left
    
    // convert top and bottom lat in mercator to degrees
    // note that in fact the coordinates go from about -85 to +85 not -90 to 90!
    double lat = (180 / Math.PI) * ((2 * Math.Atan(Math.Exp(Math.PI * (1 - (2 * latMerc)))))
                       - (Math.PI / 2));
    return new Point(lng, lat);
}

By the way, feel free to use any of the code in this post in your own application. Consider it "Public Domain". However, I would appreciate (but not require) if you would place a comment near it indicating that this blog is the source of the original code.

Hopefully this helps those of you writing mapping applications in C#! Please post any questions or comments in the comments section!

Writing a blog for fun and profit... I mean grades

andrew@andrewnurse.net — Thu, 08 Jan 2009 00:00:00 -0800

The Spring semester at SFU is off to a running start (after a day lost to the snow), and I've already got some homework to do! One of the classes I'm taking is a course titled "Writing for Computer Scientists". The goal is to develop writing skills through practice. Lots and lots of practice :).

Each class (3 times a week), we're required to submit some "writing"?. I use quotes simply to emphasize the freedom of this assignment. All I have to do is write a single word, and turn it in each week. Of course, that's not really the point of the assignment ;). One option, provided by the professor, is simply to start a blog (if you don't already have one) and post 2-3 entries per week. Given that I already have a blog which I have had trouble maintaining, I figure this is a good kick in the behind to get myself blogging again.

So, I'll hopefully be blogging more often (at least while my grades depend on it :D). I'll probably stick to the current subject of the blog: observations from an open-source developer on the Microsoft ASP.Net platform. Just in case I start to wander to different topics, I'll make sure to tag appropriately, so my normal readers can subscribe just to the posts they're interested in.

It's time to start blogging for fun and ~~profit~~ grades!

Playing with Azure and SQL Data Services: No "Random Access Paging"

andrew@andrewnurse.net — Tue, 16 Dec 2008 00:00:00 -0800

So, I've started playing around with the whole Azure Services Platform (http://www.azure.com), now that school's done :) and I thought I'd blog about things I discover along the way.

My first idea was to create a simple blogging engine in MVC that would store data in SQL Data Services and run in Windows Azure (I got my invite code recently, yay!). However, I soon discovered that would be difficult. It turns out that while SQL Data Services does support "Paging" the results of a query so that you don't retrieve every result in one batch, it does not support what I refer to as "Random Access Paging". Random Access Paging (and I'm sure theres some other term for it) basically means providing a page index and page size to the data layer and having it retrieve just that page (i.e. Skip/Take in LINQ). There are some solutions, but most of them requrire that you step through each page sequentially, rather than just being able to request a page.

Of course, I may be missing something. So, if you know how I could solve this problem, feel free to post in the comments!

In a blogging engine, this is practically a necessity, since you only want to display a certain number of posts per page. So I've tabled that idea for now.

My next idea: An RSS reader which stores your set of subscribed feeds in your Live Mesh (though I don't have access to that CTP yet).

Backing up a Zune Library for when you reinstall Windows

andrew@andrewnurse.net — Thu, 11 Dec 2008 00:00:00 -0800

I regularly reinstall Windows, so when I used iTunes, I had to make sure I knew how to back up my music library (because I didn't want to re-rate all my songs :D). I recently switched back to Zune from iTunes because I got so frustrated with my cracked iPod Touch screen. Then, came the invevitable reinstall :)

I keep my music on a separate partition, along with ALL my data (Folder Redirection for the win!), so the music files aren't the problem. With iTunes I could place the music library database on the other drive too, but so far no luck there with the Zune software (I'm going to check out Junctions to see if that will work). So, I resigned myself to digging through my AppData folder to see if I could find the stuff I needed.

For those who don't already know, the AppData folder, in Windows Vista is in: C:\Users\[UserName]\AppData. This is where programs (should) store their per-user settings and configuration data. Unfortunately (AFAIK) it's not possible to redirect this folder (like you can with Documents, Music, etc.).

The Zune software stores its settings in C:\Users\[UserName]\AppData\Local\Microsoft\Zune. All I had to do was copy this folder over to my second partition, reinstall the OS and reinstall the Zune software. Then, I started the Zune software, just to be sure and shut it down again. I copied the backed up folder back into place, restarted the Zune software and voila my library is back. I was even able to reconnect my Zune and have it automatically recognize it and sync!

So, in summary:

Back up C:\Users\[UserName]\AppData\Local\Microsoft\Zune
Reinstall Windows and other Apps
Reinstall Zune
Start Zune software (not sure if it's necessary, but it worked for me)
Shutdown Zune software
Copy backup back in to C:\Users\[UserName]\AppData\Local\Microsoft\Zune
Start Zune software and enjoy!

Let me know if that works for you, or if you have problems!

VidSpeak Part 3 - Generating and Playing Audio

andrew@andrewnurse.net — Fri, 21 Nov 2008 00:00:00 -0800

This is part 3 of my VidSpeak series, where I show off an app I wrote for my multimedia course. Check out part 2.

5. Generate the Sound

Alright, so this is the complex part :). For the assignment, we were instructed to convert each column of the quantized grayscale 64x64 frame to a chord of sound. We were asked to do this by assigning a frequency to each row of the image (the middle row being 440Hz or "A440", which is the "A" above middle "C" on a piano). Then the "brightness" of the pixel at that row is the intesity of that frequency in the chord. So, if a pixel was black, that frequency was not part of the chord, and if it was white, that frequency was part of the chord (at 100% intensity).

Sound is generated using a process called Frequency Modulation (or "FREAK WHEN SEE MOD YOU LATE SHUN" as I like to call it (inside joke for MS interns who did The Microsoft Intern Game :P)). Essentially what we are going to do is create sine waves of each frequency we want to play, scale them by the intensity value, and then add them together to create a new sine wave, representing the whole chord. This process is done with this code


uint* data = (uint*) Marshal.AllocHGlobal(SamplesPerColumn * Marshal.SizeOf(typeof (uint)));
for (int sample = 0; sample < SamplesPerColumn; sample++) {
    double signal = 0.0f;
    for (int col = 0; col < width; col++) {
        PixelData* cell = row0 + col;
        double intensity = cell->Red / (double) (_quantizationLevels - 1);
        signal += (intensity * Math.Sin(2.0f * Math.PI * FrequencyTable[col] * TimeTable[sample]));
    }
    signal = (uint.MaxValue / 2) * (signal + 1);
    *(data + sample) = (uint) signal;
}

Because sound is represented digitally in computers, we can't send the sine wave itself, we have to take discrete samples of it that we will send to the sound card. In this case, I'm taking 500 samples for each column (SamplesPerColumn = 500). Elsewhere in my code I informed the sound card that I would be generating sound at 8000Hz (or 8000 samples per second), meaning that 500 samples will produce 0.0625seconds of audio.

For each sample I'm going to generate, I iterate across the columns (remember, I rotated the image in step 1 so that each row of the image now represents a column of the original frame). Then I generate a sample of the sine wave for that frequency and multiply it by the value at the pixel (as a float, where 15 is 1.0 and 0 is 0.0) (that's the line that starts "signal +=". Once I've added up all the samples for the chord, I convert the float into an unsigned integer where (uint.MaxValue / 2) represents 0 and store in the (unmanaged) buffer I'm building.

From there, I just send that unmanaged buffer 'data' to the sound card using a helper class I created to wrap the waveOut API:

SoundDevice.WriteAudioBlock(new IntPtr(data), (uint)SamplesPerColumn);

The helper class basically wraps a low-level Win32 API called "waveOut"?. It's not as powerful as using DirectX or some other high-level API, but it gets the job done :). The code for this helper is in the SoundDevice class in my code. Remember, all the code is posted in my Part 1 post.

That's it for VidSpeak. I'm going to be posting more code from my other school projects soon, but I have to finish them first :P.

Sorry about the delay on VidSpeak Part 3

andrew@andrewnurse.net — Sun, 09 Nov 2008 00:00:00 -0800

VidSpeak Part 3 is a little delayed. However, the main reason is that I've been working on a bunch of other school projects, which I plan to post about in future posts :). Stay tuned for the last part of VidSpeak and some more stuff ;)

VidSpeak Part 2 - Bitmap Manipulation

andrew@andrewnurse.net — Thu, 30 Oct 2008 00:00:00 -0700

This is part 2 of my VidSpeak series, where I show off an app I wrote for my multimedia course. Check out part 1.

To recap from last time, the problem this assignment is trying to solve is to convert video frames to some kind of audio representation. There are the steps required:

Extract the next frame from the video
Scale the frame down to 64x64 pixels
Make the frame a grayscale image
"Quantize" the grayscale frame into 4-bit colour
Convert the frame to sound

I covered extracting video frames in Part 1, so now we have to manipulate the image to prepare it for rendering as sound.

2. Scale the frame down to 64x64 pixels

All the transformation steps (2-4) are performed by "filters" which recieve a System.Drawing.Bitmap and return a transformed one (a filter could also modify the incoming Bitmap and then just return the same object). Scaling is trivial with the "GetThumbnailImage" method on System.Drawing.Image:

public Bitmap ProcessImage(Bitmap input) {
    return new Bitmap(input.GetThumbnailImage(TargetWidth, TargetHeight, null, IntPtr.Zero));
}

3. Make the frame a grayscale image

There may be a function in .Net to do this, but as part of the assignment we were provided with a formula to convert RGB images to 255-shade grayscale images so I decided to do this. Now, the Bitmap class provides GetPixel and SetPixel methods, but they aren't really very performant when you need to touch every pixel. Fortunately, the Bitmap class provides a method called "LockPixels" which prevents the Garbage Collector from moving the Bitmap data around in memory. With that, I made a base class for filters that process a Bitmap on a Pixel-by-Pixel basis:


BitmapData data = input.LockBits(new Rectangle(0, 0, input.Width, input.Height), 
                                 ImageLockMode.ReadWrite, 
                                 PixelFormat.Format24bppRgb);
unsafe {
    for (int col = 0; col < data.Width; col++) {
        for (int row = 0; row < data.Height; row++) {
            PixelData* pixel = (PixelData*)data.Scan0 + col + row * data.Width;
            ProcessPixel(pixel);
        }
    }
}
input.UnlockBits(data);

PixelData is a simple 3-byte struct:

[StructLayout(LayoutKind.Sequential)]
public struct PixelData {
    public byte Blue;
    public byte Green;
    public byte Red;
}

Then, my grayscale image filter just has to implement ProcessPixel


byte convertedValue = (byte) Math.Floor((0.299 * pixel->Red) + 
                                        (0.587 * pixel->Green) + 
                                        (0.114 * pixel->Blue));
pixel->Red = convertedValue;
pixel->Green = convertedValue;
pixel->Blue = convertedValue;

Kinda strange to see the "->" operator in C#, eh?

4. "Quantize" the image to 4-bit colour

Quantization is the process of "compressing a range of values to a single quantum value" (http://en.wikipedia.org/wiki/Quantization_(image_processing)). In this case, we are quantizing an 8-bit Grayscale image to a 4-bit Grayscale image. To do this, we take each pixel value, and assign it one of 16 partitions (4-bits can hold values 0 through 15). The following code (in FixedPartitionQuantizationStrategy) does this (note _partitionSize is 16 in this case).

public byte Quantize(byte input) {
    return (byte)Math.Floor((double)input / _partitionSize);
}

Quantization is done by the QuantizeImageFilter, which uses a similar base class to the Grayscale conversion above. Then, an IQuantizationStrategy is used to perform the Quantization itself:

protected override unsafe void ProcessPixel(PixelData* pixel) {
    pixel->Red = _strategy.Quantize(pixel->Red);
    pixel->Green = _strategy.Quantize(pixel->Green);
    pixel->Blue = _strategy.Quantize(pixel->Blue);
}

And thats it! We now have an image that is 64x64 pixels in 4-bit Grayscale. Next we have to render it out to sound. Stay tuned for Part 3 for that!

VidSpeak Part 1 - Extracting Frames from Video in C#!

andrew@andrewnurse.net — Tue, 21 Oct 2008 00:00:00 -0700

I know, its been too long since I blogged, but its pretty busy at school right now :). Anyway, I'm taking a course in Multimedia this semester, and as part of that course, I have to write a program to convert frames in a Video to short Audio clips. I thought it might be interesting to examine how that is done, in C#. So, over the course of about 3-4 posts, I'll go over the code that I wrote. I've attached the full project to this post, so you can take a look at it right now. The GUI app should work, though I can't guarantee it. All I can give it is "Works on My Machine" seal of approval :)

Here are the steps involved:

Extract the next frame from the video
Scale the frame down to 64x64 pixels
Make the frame a grayscale image
"Quantize" the grayscale frame into 4-bit colour
Convert the frame to sound

The Code

The Code is a Visual Studio 2008 solution, written for .Net 3.5. It uses unsafe code for image processing and sound generation, so you can't run it without full trust (i.e. you can't run it off of a network share).

1. Extract the next frame from the video

I used a "Pipeline" (http://en.wikipedia.org/wiki/Pipeline_(software)) architecture, so this phase is handled by a component I call a "Frame Source" which is expected to return a new frame when asked (or return null to signal the end of the input). I used the DirectShow COM library "DexterLib" to do the extraction. DexterLib contains a class called MediaDet (for MediaDetector) which does most of the work. Here's the code for the function which retrieves a frame at a specified timecode (in seconds).

FYI: "_detector" is an instance of DexterLib.MediaDetClass() ("_detector" is of type IMediaDet), "_streamLength" is the length of of the video stream in seconds, "_bufferHandle" is an IntPtr referring to an unmanaged buffer (allocated with Marshal.AllocHGlobal) to hold the bitmap, and "_bufferSize"/"_frameSize" are the size of the buffer and the size of each video frame (respectively)

// WARNING: This method will destroy the bitmap retrieved in a previous call to this method
public Bitmap GetFrameAtTime(double timeCode) {
    // Get the bitmap at this time
    Bitmap frame = null;
    unsafe {
        byte* bufferPointer = (byte*)_bufferHandle;
        _detector.GetBitmapBits(timeCode, 
                                ref _bufferSize, 
                                ref *bufferPointer, 
                                _frameSize.Width, 
                                _frameSize.Height);
        frame = new Bitmap(_frameSize.Width, // Width
                           _frameSize.Height, // Heigth
                           _frameSize.Width * 3, // Stride
                           PixelFormat.Format24bppRgb, // Pixel Format
                           new IntPtr(bufferPointer + 
                                      Marshal.SizeOf(typeof(BITMAPINFOHEADER)))); // Start of Buffer
    }
    return frame;
}

(Note: If you look at the actualy code, you will notice I snipped out some stuff from the beginning of this function to display it on the blog. The missing code just handles an (experimental) feature I added to allow me to start at any location in the video, rather than always starting at the beginning)

After loading the frame, I have to flip it, because Dexter loads the frame upside-down, fortunately the System.Drawing.Image class provides a RotateFlip method to do just that! I also rotate it 90 degrees clock-wise, so that each row of the transformed image maps to a column of the frame. This makes step 5 easier, since Bitmaps are stored in "row-major" order (http://en.wikipedia.org/wiki/Row-major_order).

To use the Frame Source, all my program has to do is call GetFrameAtTime method passing in a timecode (in seconds). This is handled in the FrameProcessor by the GetNextFrame method

_source.GetFrameAtTime((DateTime.Now - _startTime).TotalSeconds)

Rather than going frame-by-frame, I'm extracting the next frame by time. So, if it takes 4 seconds to process a frame, the next frame I take is approximately 4 seconds after the frame I just processed.

Here's the code: VidSpeak.zip (267.21 KB)

Fun Summer

andrew@andrewnurse.net — Sat, 23 Aug 2008 00:00:00 -0700

Well, its been a fun summer, but my time at Microsoft is over ...

... for now :). It looks like I'll be coming back full-time come next August, so it's a bittersweet end. I'll miss the people on my team and the interns I've met, but in most cases I'll see them again next year! Keep an eye on this blog, because you can bet I'll be posting when the stuff I worked on goes live!

Awesomeness, in a box.

andrew@andrewnurse.net — Tue, 19 Aug 2008 00:00:00 -0700

Thanks to Alex Tuteur for typing up the lyrics from our scribbles on
paper, to Kyle Farnung for filiming, converting and posting the
video, to Kristine Johnson for singing the epic win-ness, and to all the people who helped write the song!

STILL AROUND
By Andrew Nurse, Kristine Johnson, Alex Tuteur, Ben Butler, Kyle Farnung, Steven Truong (EXP), Adam Kiu, and William Hong

Internz: a triumph
I'm sending a lolcat: EPIC FAIL
It's hard to understate procrastination
MICROSOFT INTERNS
We do nothing but read mail all day
For the good of all of us
Except for the FTEs
But there's no sense crying over broken compiles
You just keep on building, it'll work in awhile
And the coding gets done
And you make a bad pun
For the interns who are
STILL AROUND

I'm not really working
I'm reading Internz all day long
Even though I didn't meet commitments
And now it's review time
And it is my future on the line
As I leave it sucks because
I was so happy right here
Now it's time to leave and I am packing my bag
To go back to my school and deal with the jet lag
So I'm GLaD that I've turned
Think of all the cash we've earned
As the interns who are
STILL AROUND

I'm going to leave now
I guess you prefer to stay right here
Maybe I'll find someone else to work for
Maybe at Google
THAT WAS A JOKE, HAHA, FAT CHANCE
Anyway this lolcat's great
It's so amusing and fun
Look at me still coding when there's packing to do
When I look out there it makes me GLaD I'm not you
I've got all my boxes done
Now it's time for me to run
Bye bye interns who are
STILL AROUND

I can't believe you are all
STILL AROUND
I'm done with coding and you're
STILL AROUND
I'm leaving now and you are
STILL AROUND
While I'm at school you'll be
STILL AROUND
And when I'm back here you'll be
STILL AROUND
STILL AROUND
STILL AROUND

Rendering difference between Firefox and IE when using innerHTML

andrew@andrewnurse.net — Sat, 16 Aug 2008 00:00:00 -0700

I know, its been a while since I've blogged, but I'm just so busy with my Microsoft internship its been hard to find time. Anyway, on to the post...

I've been doing quite a bit of JavaScript work recently and I discovered an interesting inconsistency between Firefox and IE when it comes to the innerHTML project. Federico, from the ASP.Net QA team blog, posted the details on that blog, but here's a quick summary.

Basically, it comes down to when the browsers perform whitespace normalization. In HTML, multiple whitespace (space, tab, newline etc.) characters are treated as a single space. Also, whitespace at the beginning of "block" elements (which are elements like DIV which have newlines before them) is removed. In Firefox, the whitespace is preserved exactly as typed when stored in Firefox's internal structures. When the HTML is rendered out, Firefox performs the normalization on the fly. However, in IE, the whitespace is normalized before the internal structures are built.

The result of this is that the contents of the innerHTML property for a DOM element can vary between these two browsers.

Check out the blog post for more details.

Off to Redmond

andrew@andrewnurse.net — Mon, 05 May 2008 00:00:00 -0700

Phew, I've been posting a lot of forum posts and blogs today trying to tidy up loose ends, because I'm off to Redmond today! I'll be going back to Microsoft for another summer internship, and I'm really excited!

I'll try to post more later in the week (because I really have to shutdown my computer and pack it up now :P) so keep an eye here, and on my Twitter feed: http://twitter.com/anurse (follow me?).

Playing with IE8: Reports Module as a WebSlice

andrew@andrewnurse.net — Thu, 06 Mar 2008 00:00:00 -0800

So, I?downloaded and installed the IE8 Beta yesterday and decided to play around with WebSlices. I found it really easy to create a simple WebSlice (just apply some CSS Classes) and I decided to try and do this with the Reports Module.

I was suprised with just how easy it was! I have already created a replacement ViewReports.ascx file that turns every Reports Module into a WebSlice. I've attached the new file here, just replace your existing DesktopModules/Reports/ViewReports.ascx file with it if you want to check it out. Any Visualizer should work, but I've only tried the Grid.

CAUTION: This is for preview purposes ONLY! This replacement file has NOT been tested in ANY SIGNIFICANT?WAY (other than trying it out once). Back up your existing ViewReports.ascx file BEFORE using this!

Here's the popup you see when you view a report

And here's a screenshot of the WebSlice in action:

Anyway, the file is attached. Try it out on one of your test sites! I'll probably integrate this into an upcoming version in a more configurable way, since it's so easy :).

ViewReports.zip (.54 KB)

UPDATE: Changed to a ZIP file

Developing an RSS Data Source - Part III

andrew@andrewnurse.net — Sun, 02 Mar 2008 00:00:00 -0800

In this series of blogs, I'm going to follow the development of an RSS Data Source. I'm going to use Visual Studio 2008, but I'll be using .Net 2.0 features so you should be able to follow along in Visual Studio 2005. You can also use the Express editions of Visual Web Developer (both 2005 and 2008 versions should work). All the code will be in VB.Net, but it should be straightforward enough for C# developers to understand (after all, its all .Net)

Continued from Part II.

Part III - Settings

In this part, we're going to add a setting page so that we can change the RSS feed that our Data Source downloads.

In current versions of the Reports Module, Data Source settings are stored in the ModuleSettings table provided by DotNetNuke. This ensures that all copies of the same module instance will share the same Data Source settings (Visualizer settings are not shared between copies). However, Data Source developers need not (and should not) access these directly. Instead, the Reports Module provides a layer on top of the ModuleSettings table called "Data Source Settings". When the Data Source is executed, the Reports Module passes a Dictionary containing these settings (inside the ReportInfo object). To allow users to edit the settings, we go back to the Settings.ascx file we created in Part I.

Let's start with the user interface for our settings page. All we need for now is a place for the user to enter a feed URL. When we're done, it should look like the screenshot below

Figure 1 - The finished settings UI

So, open up the Settings.ascx file and make sure you are in Design mode. First we need to add a label, so users know what they should type in our text box. In the Solution Explorer, find the file: controls/labelcontrol.ascx and drag it on to the design surface.

Figure 2 - Locating the LabelControl.ascx file

Next drag an ASP.Net TextBox control, from the Toolbox, on to the surface. You should have something that looks like this:

Figure 3 - Locating the LabelControl.ascx file

Now, set the properties of the controls to the following values:

Property	Value for Label	Value for Text Box
`ID`	feedUrlLabel	feedUrlTextBox
`ControlName`	feedUrlTextBox	N/A
`Width`	N/A	100%

Save your changes, and open up the code file: Settings.ascx.vb. Here, we need to add code to connect our text box to the Data Source settings. The key to this is the LoadSettings and SaveSettings methods provided by ReportsSettingsBase. First, we need to import some extra namespaces:

Imports DotNetNuke.Modules.Reports
Imports System.Collections.Generic

Then, we can implement the LoadSettings and SaveSettings methods.

Public Overrides Sub LoadSettings(ByVal Settings As Dictionary(Of String, String))
    MyBase.LoadSettings(Settings)
    feedUrlTextBox.Text = SettingsUtil.GetDictionarySetting(Settings, _
                                                            "FeedUrl", _
                                                            String.Empty)
End Sub
 
Public Overrides Sub SaveSettings(ByVal Settings As Dictionary(Of String, String))
    MyBase.SaveSettings(Settings)
    Settings("FeedUrl") = feedUrlTextBox.Text
End Sub

The LoadSettings method is provided with a System.Collections.Generic.Dictionary(Of String, String) containing the current settings saved for the Data Source. Our implementation uses the SettingsUtil helper class (provided with the Reports Module) to retrieve a value from the dictionary, or return a default value if it doesn't exist (in this case, an empty string).

The SaveSettings method is also provided with the current settings. However, this method is responsible for retrieving the values entered by the user and updating the settings.

Save the file, and navigate to the Settings page on your test module. You should see our new Settings control displayed, as in the screenshot below:

Figure 4 - Settings page so far

Oops, we still need to put the label text in! We're going to support the DotNetNuke Localization Framework, so we need to put the text in two places.

First, lets add the localized text. Create a Resource File called Settings.ascx.resx in /DesktopModules/Reports/DataSources/RSS/App_LocalResources file and open it. In Visual Studio, you get a nice table interface for editing resource strings. Add the following entries to the table (feel free to tweak the values as you want, just keep the name the same):

Name	Value
`feedUrlLabel.Text`	RSS Feed URL
`feedUrlLabel.Help`	Enter the URL to an RSS feed to retrieve data from

Your resource file should look like this after making those changes

Figure 5 - Resource File after entering values

Now, go back to Settings.ascx and set the following properties on the label we created earlier:

Property	Value
`Text`	RSS Feed URL
`Suffix`	:
`CssClass`	SubHead

Note: That's a colon (':') in the Suffix property. Also, CssClass is case sensitive.

Now, save and refresh your page. You should see something like the screen shot below:

Figure 6 - The finished Feed URL text box

Now, to change our Data Source code to use this new setting. Let's go back to the /App_Code/RSSDataSource/RSSDataSource.vb file and take a look at the signature for the ExecuteReport method:

Public Overrides Function ExecuteReport(ByVal report As ReportInfo, _
                                        ByVal hostModule As PortalModuleBase, _
                                        ByVal inputParameters As IDictionary(Of String, Object)) As System.Data.DataView

The important parameter here is the report parameter. There is a property called DataSourceSettings on that object which contains the same dictionary we created in SaveSettings. First, delete the FeedUrl constant we were using before. Then add following code to the beginning of the ExecuteReport method to get the Feed URL from the settings:

If Not report.DataSourceSettings.ContainsKey("FeedUrl") Then
    Throw New RequiredSettingMissingException("FeedUrl", MyBase.ExtensionContext)
End If
Dim feedUrl As Uri = Nothing
If Not Uri.TryCreate(report.DataSourceSettings("FeedUrl"), UriKind.Absolute, feedUrl) Then
    Throw New RequiredSettingMissingException("FeedUrl", MyBase.ExtensionContext)
End If

This code checks for the setting, and if it isn't present or if it isn't a valid URL we throw an exception provided by the Reports Module: RequiredSettingMissingException. We pass it the name of our setting and some contextual information about our Data Source (which is provided automatically by our base class. If the setting isn't present, the Reports Module will automatically display a useful error message indicating that the setting is missing.

While we're in the code, let's add HTML Decoding directly to the Data Source, so we don't have to use the HTML Decode converter in the Module Settings. To do that, we change the line that adds entries to the output table so that it automatically HTML Decodes the description:

dt.Rows.Add(title, New Uri(link), HttpUtility.HtmlDecode(description))

Make sure everything is saved and go back to the website. You should probably go back to the home page, just to make sure everything is properly recompiled. Go back to the settings page for the Reports Module, and make sure the RSS Data Source is selected. Then configure it with your favourite RSS feed. This time, I'll use my personal blog's RSS feed (WARNING: Shameless plug alert!).

Figure 7 - Testing the Settings UI

Once you've done that, make sure the HTML Decode property is set and the HTML Visualizer is properly configured, just like in part 2 and click Update. You should see the RSS feed displayed just like in Part 2, only now we can change the URL!

Figure 8 - The Final Results

Conclusion

At this point, we have a working RSS Data Source! You can stop here if you want, but in the next part I'll cover packaging the Data Source up so that it can be installed in any Reports Module installation.

Download the code so far.

Developing an RSS Data Source - Part II

andrew@andrewnurse.net — Fri, 29 Feb 2008 00:00:00 -0800

UPDATE: Part III is now available.

Continued from Part I.

Part II - WebRequests, XPath, and Data Tables, oh my!

In this part, we're going to add the code to retrieve data from the RSS feed. First, we need to create a folder to hold our code. We're going to put our code in the App_Code folder, so it will be compiled automatically by ASP.Net. That way we can save a few extra steps and get straight to the point. You could also put this code in a separate .Net Class Library, and put that library in the Bin folder of your website.

We'll create a sub-folder inside the App_Code folder for our code and call it RSSDataSource:

Figure 1 - Creating an App_Code folder

Next, we need to tell ASP.Net to compile code found in this folder. Open the web.config file, in the root of the website, and find the <codeSubDirectories> section (its inside the <system.web> section). In my version of DotNetNuke it looks like this, but yours may have different <add> entries:

<codeSubDirectories>
  <add directoryName="HTML" />
</codeSubDirectories>

We need to add an entry to this list for our new subfolder. On my system, it looks like this (again, yours may vary slightly):

<codeSubDirectories>
  <add directoryName="HTML" />
  <add directoryName="RSSDataSource"/>
</codeSubDirectories>

Now that we have somewhere to put our code, let's create our Data Source! We'll create a class inside our nice new App_Code folder called RSSDataSource.

I'm not going to list the code for the Data Source in this post, because I've attached it to this post. However, I'll go over the general steps.

First, we use the System.Net.WebClient class to download the RSS feed. For now, I've hardcoded it to one of my favourite blogs: Scott Hanselman's Computer Zen
Next we create an ADO.Net DataTable with three columns: Title, Link and Description
Then, we use the classes in the System.Xml.XPath namespace to add a row for each item in the feed to the DataTable.
Finally, we wrap the DataTable in a DataView. The Reports Module expects us to return a DataView in order to allow us to do filtering and sorting if we wanted to. We aren't using that feature, so we just create a simple DataView to wrap our table.

There's only one last step to hook everything up. We need to tell the Reports Module how to find our Data Source code. To do that, open the code-behind file for DesktopModules/Reports/DataSources/RSS/Settings.ascx and change the DataSourceClass property to the following code:

Public ReadOnly Property DataSourceClass() As String Implements IDataSourceSettingsControl.DataSourceClass
    Get
        Return GetType(RSSDataSource).FullName
    End Get
End Property

And with that, we have a working RSS Data Source! Go back to your browser and go to the Settings page for the Reports Module instance we created in Part 1. Select the RSS Data Source as the Active Data Source, and click update. You should see a very wide grid containing the data from the RSS feed (assuming you are using the Grid Visualizer, which is the default):

Figure 2 - Simple Grid View

Well, that doesn't look very useful does it? Let's start by getting rid of that HTML junk in the Description field. To get rid of that, go back to the Settings page, and add "Description" to the list of columns to HTML Decode.

Figure 3 - HTML Decode Converter

That should clean up the HTML, but it's still a big grid. Let's use the HTML Visualizer to clean it up a bit. Put the following code in an HTML file and place it in your Portals/[PortalID] folder:

<h1><a href="[Link]">[Title]</a></h1>
[Description]
<hr />

Then, go back to the Settings page, select the HTML Template Visualizer and the HTML file you just uploaded:

Figure 4 - Configuring the HTML Visualizer

Click Update and you should see a much more readable display. It's almost like a real RSS Reader!

Figure 5 - A Reports Module-powered RSS Reader!

Conclusion

Well, that's all for part two. Next, we'll cover Data Source settings so that we can point our Data Source at any RSS feed we want, rather than hard coding it. Then, we add an HTML Decoding feature directly into the Data Source so we don't have to configure the module to decode the Description field. Finally, we'll package it all up so that others can download and install it.

Download the code so far.

Developing an RSS Data Source - Part I

andrew@andrewnurse.net — Fri, 29 Feb 2008 00:00:00 -0800

UPDATE: Part II is now available.

Part I - Setting Up

First things first, you'll need to install a Source distribution of DotNetNuke. I'm not going to cover that here because there are some other resources out there. You'll also need to install the Source package of the Reports Module. Once you've done this, open your DotNetNuke website up in Visual Studio

Figure 1 - Open Website Dialog

After opening it, expand the DesktopModules/Reports/DataSources folder. You should see something similar to the following

Figure 2 - Data Sources Folder

Each of the folder under the DataSources folder represents a different Data Source. So, lets get started and create a folder for our Data Source and call it RSS. Inside that folder, we need to create an App_LocalResources folder. Visual Studio provides a special menu option to do that:

Figure 3 - Add App_LocalResources Folder menu item

Inside there, we must create a Resource file. Even if you aren't planning to translate your Data Source into different languages, the Reports Module uses this file to determine the name of your Data Source, so you must create it. In this sample, we aren't going to use the DotNetNuke Localization framework, so this is the only time we'll need to delve into Resource files. Add a new Resource file to the App_LocalResources folder called "DataSource.ascx.resx".

Figure 4 - Adding a Resource File

Open this file and add a new resource key called "DataSourceName.Text" with a value of "RSS".

Figure 5 - Editing the Resource File

We've almost got a running, albeit useless, Data Source. There's only one more file to add. Back in the "RSS" folder, add a Web User Control called "Settings.ascx". Open the Code-behind file (Settings.ascx.vb) and replace the contents with the following code:

Imports DotNetNuke.Modules.Reports.Extensions
Imports DotNetNuke.Modules.Reports.DataSources
 
Partial Class DesktopModules_Reports_DataSources_RSS_Settings
    Inherits ReportsSettingsBase
    Implements IDataSourceSettingsControl
 
 
    Public ReadOnly Property DataSourceClass() As String Implements IDataSourceSettingsControl.DataSourceClass
        Get
            Return String.Empty
        End Get
    End Property
End Class

Save everything and open your Web Browser. Add an instance of the Reports Module to a page and open the Settings page. You should now see the "RSS Data Source" in the Active Data Source drop-down. You can select it, but it doesn't do anything, so don't click Update.

Conclusion

That's all for part one. I know there isn't anything really useful yet, but I want to keep these parts short. Tomorrow, we'll add the code to retrieve data from an RSS feed.

Download the code so far.

Just signed my offer letter

andrew@andrewnurse.net — Wed, 13 Feb 2008 00:00:00 -0800

Well, I just signed my offer letter so I'm going back to Microsoft for another summer! This time, I'll be working on the ASP.Net team, which is exciting. It's going to be a lot of fun having a chance to contribute to a framework I've been using for the past 2 years (I've almost forgotton what a Windows Form is :P).

I just thought I'd share that exciting news with my readers. However, I do have a few other updates:

The Reports Module 5.0 User Guide is almost "content-complete" so I'll just need to get it formatted properly and it should be ready to go live
I'm starting a Blog series on Developing for the Reports Module. For now, this is going to have to stand in for a full Programming Guide, but hopefully it will be a good starting point for those of you thinking of writing Visualizers or Data Sources.

Anyway, look for my first post in that series soon. I'll be crossposting it on my DotNetNuke blog, so feel free to check it out there.

T-11 hours till my Interviews begin

andrew@andrewnurse.net — Fri, 18 Jan 2008 00:00:00 -0800

So, I'm chilling in my hotel in Redmond watching MythBusters (best show ever!) while I await the start of my Microsoft Interviews at 8am tomorrow morning.

Now, on to things my readers would be interested in :). I'm going to start a series on Extending the Reports Module in the next few days that's going to start by developing a custom Data Source for the module that extracts data from an RSS feed. I'll probably move on to developing a custom Visualizer and even more tutorials.

I'm also going to try to set up a documentation Wiki for the module and pre-populate it with some basic documentation for Reports 5.0.

Anyway, now that the holidays are over, I'll be getting back into blogging regularly, so keep an eye on your RSS readers!

Using Parameters in the Reports Module

andrew@andrewnurse.net — Wed, 19 Dec 2007 00:00:00 -0800

A common question that comes up in the Reports Module forums is "How can I display data relevant to the current User in the Reports Module?". So, I figured I should write a blog post about a feature that has been in the Reports Module for a few versions. When writing your SQL Scripts for the Reports Module, you automatically have access to four parameters: @PortalID, @TabID, @ModuleID and @UserID. When the SQL Script is executed, these are replaced by the ID of the current Portal, Tab, Module and User (respectively).

So, if you wanted to display a list of Roles for the currently logged in user you could use this script:

SELECT R.* 
FROM   {oQ}UserRoles UR 
   INNER JOIN dnn_Roles R 
   ON       UR.RoleID = R.RoleID 
WHERE  UR.UserID = @UserID

When you run this script, the current user's ID will be substituted in the place of the @UserID parameter. (Note: This is done by SQL Server itself, which avoids SQL Injection attacks). Also, the {oQ} token is replaced with the Object Qualifier configured for your database (if you configured one in your web.config file)

Here's another example. It's a script that retrieves the Website profile property for each user in the current portal.

SELECT    U.Username,
         (SELECT    PropertyValue
          FROM     {oQ}UserProfile AS UP
          INNER JOIN {oQ}ProfilePropertyDefinition AS PPD 
          ON       PPD.PropertyDefinitionID = UP.PropertyDefinitionID
          WHERE    UP.UserId = U.UserId
          AND       PropertyName = 'Website'
          AND      PropertyCategory = 'Contact Info') AS 'Website'
FROM     {oQ}UserPortals AS UP
INNER JOIN {oQ}Users AS U 
ON       UP.UserId = U.UserId
WHERE    PortalID = @PortalID

The italicized section is the part that actually retrieves the property value, you can replace the PropertyName and PropertyCategory values as you like and repeat this section multiple times to bring back multiple profile properties.

One of the planned future features for the module is to support user-specified parameters. This means you could place a text box, drop down, calendar or other control on the view page for the module and the value of this control would be used as a parameter to the Report. At the moment you're limited to these four parameters, but there's still a lot you can do with them!

Reports Module v5.0 Released to Tracker

andrew@andrewnurse.net — Wed, 19 Dec 2007 00:00:00 -0800

After a few delays, version 5.0 of the Reports Module has been released to the Module Release Process. This means that it should be released shortly (though given the time of year, it may take a couple of weeks). I'll be posting some How-To screencasts walking through the new features (and some old ones too) in the next week or so. I'm also working on a full User's Guide to be released shortly.

One important note regarding Reports 5.0: The module requires at least v4.6 of the DotNetNuke core, it will NOT run on a previous version of the core. This is because of the new Visualizer and Data Source installers which use the Package Installer introduced in v4.6. I hope to release a version of the module with the installer portions removed which will be referred to as "Version 4.9".

To give you an idea of what's coming, here's an excerpt from the release notes containing a list of resolved issues in this version (The same list can be found here: http://support.dotnetnuke.com/project/RoadMap.aspx?PROJID=36)

RPT-5634 JavaScipt error when first configuring Reports DNN _04.05.01
A JavaScript error was occuring on the main settings page, this has been corrected
RPT-5676 HTML Decoding in XSL Visualizer
Fix through RPT-6514
RPT-4698 Clean up interface for Reporting Services Visualizer
Interface for Reporting Services Visualizer has been improved significantly the Visualizer has also been renamed to the "Microsoft ReportViewer Visualizer"
RPT-5715 Include {databaseOwner} token in upgrade SQL scripts
Scripts have been corrected
RPT-5791 Run Reports on Demand
Reports can now be run "on-demand" rather than automatically on load
RPT-5806 Support the EnableExternalImages property of ReportViewer control
The EnableExternalImages and EnableHyperlink properties are now supported, along with many others (see RPT-4698)
RPT-3994 Alternate Data Sources
Data Sources Framework allows Reports to run queries against other Databases (SQL Server, Access, Oracle, etc.)
RPT-6466 Disable the Use of Parameters while Caching is enabled
Paramaters are disabled when Caching is enabled due to a serious privacy risk
RPT-6514 HTML Decoding and Encoding of fields retrieved from database
Fields can now be individually HTML Encoded and HTML Decoded before being passed to the Visualizer
RPT-6516 Report Info Pane
An info pane displaying the title and description of the report can now be displayed on the main screen
RPT-6454 Allow Users to specify custom CSS Styles and Classes for the Grid in the Grid Visualizer
Users can use custom CSS to style the Grid Visualizer
RPT-6438 Add a setting to enable/disable Grid Lines in Grid Visualizer
Grid Lines can now be configured in the Grid Visualizer

Keep an eye on the blogs for more updates, I'll make sure to post here when the module is released!

Technorati Tags: DotNetNuke,Reporting,ASP.Net

More Better Familiarize Yourself with a Toolkit? (Name in progress) - #1

andrew@andrewnurse.net — Sat, 15 Dec 2007 00:00:00 -0800

While attempting to avoid infringing on the name from a segment from a popular internet audio talk show, I had an idea for a "quick tip" when I was reading a recent .Net Tip of the Day.

Throughout the product development cycle, occasionally certain methods become obsolete. If you can't modify those methods, will need to write another implementation of the method using a slightly different name or signature. To maintain compatibility, you do not want to remove the old method and break your code. This is where the .NET Obsolete attribute comes in handy:
[Obsolete("Use the new LogRequestEx instead.")]
public static void LogRequest(string feedUrl, string referer)
{
    ...
Setting the Obsolete attribute as above makes a warning message appear in the Visual Studio's Error List stating that the particular call to a method is obsolete. The warning message also includes your personalized message that you pass as the attribute's argument (such as, "Use the new LogRequestEx instead").

So, I figured I'd start of this semi-frequently series of .Net tips with an extension to this one :). Not only can you have Visual Studio display a warning, but if you really don't want people using the method you can configure it to display an error message if the user tries to use the method. For example, in the example above you need only add a boolean value for the second constructor parameter "error":

[Obsolete("You must use the new LogRequestEx instead.", true)]
public static void LogRequest(string feedUrl, string referer)
{
}

More details can be found on MSDN

Applying the Single Responsibility Principle at a Product level

andrew@andrewnurse.net — Thu, 13 Dec 2007 00:00:00 -0800

One of the big concepts in Object Oriented Programming, is the Single Responsibility Principle (SRP), which essentially states that every object in a computer program should have a single responsibility. I recently started thinking about where OneQuery fits in to the whole ORM landscape. OneQuery is designed for a very singular purpose. It is a Database-agnostic Query engine. That alone is its responsibility and I think it should stay that way. So, in fact, OneQuery doesn't fit into the ORM landscape, because it isn't an ORM. OneQuery is directly focused on the problem of writing queries against any database and receiving tabular results (i.e. IDataReader).

Does this mean that you should ignore it if you are looking for an ORM? No, I don't believe so. Just because OneQuery isn't an ORM, doesn't mean it can't work with ORM tools. One of the ideas sitting at the back of my head is to create an ORM called "OneMapper" which would sit on top of OneQuery and add ORM features. The fundamental difference between the combination of OneMapper and OneQuery and other ORM tools is that you could use OneQuery without OneMapper. Tools like SubSonic and NHibernate have a tight coupling between their ORM components and their database query components. The point is, I see OneQuery as being an example of SoC/SRP being applied at a "Product" level, rather than at a Class level. I want OneQuery to be a really good query engine, not a pretty good ORM and I think that separating those concerns makes that much more possible.

The main problem I was thinking about when I dreamt up OneQuery is how DotNetNuke could achieve true Database-agnosticism in its Data Access Layer. One option would be to use a tool like NHibernate or SubSonic to completely rewrite the Data and Business Layers with a true ORM. However the problem there is that it would be a wholesale change and would probably introduce some major breaking changes.

Another option would be to use something like SubSonic's Query Tool (which actually inspired me to develop OneQuery), which allows you to build database-agnostic queries and execute them, receiving tabular data in return (rather than objects). Then, the existing DotNetNuke infrastructure could handle the ORM stuff. This would basically mean replacing the existing "SqlDataProvider" with a "SubSonicDataProvider" and basically avoiding any breaking changes by just scooping out the old SQL Server-specific guts and injecting a tasty cream...oops, I mean Database-Agnostic...filling.

So, why not do that? Well, I realized that SubSonic is, first and foremost, an ORM and it will always be focused on that. The Query Tool is a great feature, but it feels like it's just tacked on to the main ORM component. This has two ramifications: First, the ORM will be the primary focus of development and new features in the Query Tool will be fewer and far between. And second, if you just want the Query Tool, you have to bring along the entire ORM system as well (unless SubSonic does some refactoring). At 644KB, SubSonic isn't huge but it isn't tiny either (though, I don't have exact numbers on what portion of that is the ORM and what portion is the Query Tool).

So, that leads me into the primary difference between OneQuery and other tools like NHibernate and SubSonic: OneQuery is not actually an ORM! Its a query engine, and hopefully it will be a darn good one too. That is its primary responsibility and any other cool features (like ORM components or LINQ support) are totally separate concerns.

Now, having said all that, this is just what I've been thinking about recently. Maybe it's all crazy talk. Feel free to sound off in the comments and tell me that :).

OneQuery - A Fluent Interface to Database Queries

andrew@andrewnurse.net — Tue, 11 Dec 2007 00:00:00 -0800

Aside from my school work and my work on the DotNetNuke Reports Module (very very close to release now), I've been tinkering with another project over the past few months. It's called OneQuery and its yet another Data Access Layer tool. OneQuery is designed to provide a "fluent interface" to database queries.

What is a fluent interface? Wikipedia defines it as "an object oriented construct that defines a behaviour capable of relaying the instruction context of a subsequent call". Essentially, it means that instead of writing code like this:

List<int> ints = new List<int>();
ints.Add(1);
ints.Add(2);
ints.Add(3);

You can write code like this

List<int> ints = new List<int>().Add(1).Add(2).Add(3);

That's quite a savings and, in my opinion, it's much more readable.

So, what's the point?

Well, recently I've been thinking about the issue of database agnostic...ness (database agnostivity? database agnosticism?). I really want to be able to write data access code that will run on multiple database engines without having to manually translate it. DotNetNuke provides a, theoretically, database agnostic interface through Data Providers. However, to support another database engine, the core DataProvider must be translated, Stored Procedures must be converted and all sorts of "Porting" issues arise. Wouldn't it be nice if you could WORA (Write Once, Run Anywhere... a Java term) your SQL Scripts?

Wait, hasn't this been done?

Of course it has, and many many times. First, there's the elephant in the room, LINQ. However, aside from requiring .Net 3.5, if you want to write database agnostic code, you need to use something like the ADO.Net Entity Framework. OneQuery is designed to work in .Net 2.0, and I'm planning to write a LINQ layer on top of it so that you can write LINQ queries and run them on multiple database engines (not just SQL Server). I think OneQuery could serve as a "migration path" where you can write OneQuery syntax and then move to LINQ syntax when you're ready to move to .Net 3.5 while still keeping the database-agnostic OneQuery core.

There are a lot more database libraries out there as well. SubSonic provides an excellent Object/Relational Mapper (ORM) that allows you to work directly with objects, but sometimes you want a little more control. There are, of course, many others such as NHibernate, IBATIS.Net, and EntitySpaces. However, I found that while working with these tools I would find little things that would annoy me. For example, SubSonic does provide a way for you to directly query the database and get an IDataReader back, but it has an interface I found a little awkward (though it is a nice fluent interface). There are also smaller annoying features such as a lack of support for certain SQL statements (like "INSERT") and a lack of JOIN support. On the other side, tools like NHibernate are really designed to allow you to ignore the database completely, which is a nice feature, but sometimes I need that direct database control.

So, what is OneQuery?

[Wow, I started two sub-headings off with 'so', perhaps that summer at Microsoft is affecting my patterns of speech. They say Microsofties tend to start there sentences with "so" and say "drinking from a firehose" and "super" a lot]

OneQuery is designed to let you focus on writing one query, and allows you to run the query on multiple database systems. Instead of using strings, which the compiler ignores, I wanted to provide a fluent interface and use operator overloading to make the queries as readable as possible.

Enough already, get to the code!

Ok, so here's a sample query using OneQuery's fluent syntax:

(I posted it as a screenshot to make sure the spacing worked out, I'll post the text at the end if you want to copy it).

There are a few things I want to do to let you clear that up even further. For example, it would be much easier if you could write ".OrderBy(Product.ListPrice).Desc()", or ".Where(Product.ListPrice.Between(10).And(100))"

Behind the scenes, the only other code you need to "write" is a helper class for the products table. I used quotes there because I'm planning on creating a code generator to do that for you.

CAN HAZ CODE?

Hopefully I've tantalized you a little bit and you'd like to check it out. You can download a preview release from our CodePlex Project Site, and discuss future features there as well. I'm planning a whole bunch of new features including (but definitely NOT limited to):

Support for ALL SQL DML Statements (INSERT, UPDATE, DELETE)

Full support for SELECT (including JOIN, GROUP BY, HAVING, and sub-queries)

A Code Generator to create the helper classes

and more!

Please let me know what you think! I want to make sure this system solves problems for as many people as possible, and the best way to do that is to get contributions from the community as early as possible. The CodePlex site has plenty of places to contribute, so please check it out!

Sample Code in Text Form

IDataReader rdr = Product.Query()
                         .Where(Product.ListPrice > 10 & Product.ListPrice < 100)
                         .OrderBy(Product.ListPrice, SortDirection.Descending)
                         .ExecuteReader();

Technorati Tags: Data Access,C#,VB,.Net,Fluent Interface,ORM

Test Post from Windows Live Writer

andrew@andrewnurse.net — Tue, 20 Nov 2007 00:00:00 -0800

Just writing a test post from Windows Live Writer. Look for a future, more detailed post about it!

Technorati Tags: Windows Live Writer

Here's a map!

ooo...fancy...

New Blog

andrew@andrewnurse.net — Thu, 15 Nov 2007 00:00:00 -0800

So, this is my first post in my new blog. Just a test post for now, but hopefully I'll put more interesting things here :)