1) I run my own Exchange Server.  It provides ActiveSync, Outlook Web Access, and Outlook Anywhere for my company’s email domains and my personal durdle.com domain.

2) I provide email addresses at durdle.com for family members (and a few people who just happen to share the family name).

Historically all those family emails have been maintained by Gradwell without going near my Exchange Server.  The durdle.com MX records in DNS pointed to Gradwell’s servers for mail delivery and I had a forwarder at Gradwell that would redirect my – and only my – messages to the Exchange Server.  For a number of reasons I needed to change this so that all email is delivered via my Exchange Server.  I made this change yesterday, discovering one problem in the process.

Email Suffixes

Gradwell’s mail server supports email suffix matching.  If you have a GMail account you may be familiar with the approach.  Say you have an email john@gmail.com.  Gmail will actually deliver mail to john+anything@gmail.com; you can add any suffix you like after the + and GMail will still deliver it to your inbox.  Gradwell support this by allowing you to place a period after the local part so you can use john.anything@example.com.  You don’t have to create these email addresses anywhere – you can just start using it and the mail server will make sure it gets to you.

This is fantastically useful because it means that when a website asks for your email address you can give out a custom throwaway email just for that site.  So when you sign up at DodgyCo.com, you give them the email john.dodgyco@example.com.  If you ever receive spam because DodgyCo sold your email address you’ll know a) who sold you out, and b) which email address to add to your spam filter.

Exchange Server

Enter Microsoft Exchange Server.  I’m currently running the 2010 edition, but the problem I’m about to describe (and solve!) exists in every recent version.  Exchange Server doesn’t support email suffixes.  Every recipient has to have an alias created on the server.  If you want to deliver to john.suffix@example.com you’d better remember to create an alias for that user, otherwise the Exchange Recipient Filter will reject the message since – as far as it is concerned – the user does not exist on the server.

Worse, Exchange doesn’t even implement any generic CatchAll functionality.  CatchAll would allow the server to redirect messages for non-existing recipients to a specific address. Smart people have created solutions to the CatchAll problem but all that does is allow you to define one mailbox for ALL non-deliverable messages to be delivered to.  Since I need to provide suffix matching to multiple users this doesn’t work for me.

Time to extend Exchange’s functionality!

Transport Agents

Transport Agents were introduced in Exchange 2007 to replace Exchange 2003’s SMTP Event Sinks and have been retained largely unchanged in Exchange 2010.  Broadly speaking when a message arrives at an Exchange Server it is moved through the transport pipeline as each SMTP event occurs.  Each SMTP event (Connect, MAIL FROM, RCPT TO, etc) can trigger an interested Transport Agent to perform some actions on the message.
The complete list of SMTP events and their sequence is described on this Exchange Server TechCenter page.

Transport Agents are simply DLLs written in managed code that fire on SMTP events.  Back when I contracted for the MOD I wrote a Transport Agent for Exchange 2007 which demonstrated that Exchange was capable of performing Dominance Checking on outgoing emails so I knew that it wouldn’t take to long to write a Transport Agent to filter incoming messages.  I was able to reuse some of that code and also made use of some of Wilbert De Graaf’s CatchAll Agent code which made the handling of a configuration file very quick to implement.  Thank you Wilbert!

Requirements

I had a couple of requirements for my Transport Agent:

1) It should allow redirection of multiple suffixes to different recipients.

2) It should allow me to ban delivery to any previously used suffixes and ensure that the sender receives a non-deliverable report.

I ended up meeting these requirements by using C# to write a Transport Agent that accepts its configuration from an XML file.  The config.xml defines multiple regular expressions to match against along with associated redirect addresses.  It also lists banned email addresses which the Transport Agent should reject.

To mimic the Gradwell behaviour which we were already making use of I needed to match firstname then a dot and then any word before the domain.  This is a fairly simple regular expression in C#, and looks like this:

^firstname+\.[a-z]+@example.com$

The config.xml takes the form:

<config>
    <redirect pattern="^john+\.[a-z]+@domain.com$" address="john@domain.com" />
    <redirect pattern="^jane+\.[a-z]+@domain.com$" address="jane.doe@gmail.com" />
    <banned address="john.spam@domain.com" />
    <banned address="jane.newsletter@domain.com" />
</config>

Here we define two patterns for redirect, one to an internal recipient and one redirect to an external SMTP address.  We also ban two known “bad” recipient addresses.

You’ll need to provide at least one redirect entry, but you’re not required to ban any addresses if you don’t want to.  Also, while redirects specifiy regular expressions to match, banned addresses are just strings which are simply compared. I leave it as an exercise for the reader to add regular expressions to the banned list!

The Code

Whenever the config.xml file is changed, the agent reloads it and parses the XML.  It puts the banned emails into a List and the regular expression patterns and redirect addresses into a Dictionary.  If any of the emails in the config appear invalid the new configuration is rejected and the agent continues to use the running config.

The agent defines a RCPT TO: handler.  This event is fired as soon as the sending MTA starts to transmit the recipient addresses. rcptArgs contains the argument we need: RecipientAddress, which we store as a lowercase string for comparison. Although the spec (RFC2821) actually requires email addresses to be case sensitive, I’ve never met an MTA that adheres to that part of the spec. So for our purposes email addresses are case insensitive and we can do all comparisons in lowercase.

public void RcptToHandler(ReceiveCommandEventSource source, RcptCommandEventArgs rcptArgs)
{
// Get the recipient address as a lowercase string.
string strRecipientAddress = rcptArgs.RecipientAddress.ToString().ToLower();

It performs two checks. First – is the recipient address in the ban list?

// Search the banned email List for the recipient.
bool exists = catchAllConfig.Banned.Exists(element => element == strRecipientAddress);
if (exists)
{
	// If found respond to the sending MTA with the reject response.
	source.RejectCommand(CatchAllAgent.rejectResponse);
 
	// No further processing.
	return;
}

And second, does the recipient address match any of the regular expressions?

// For each pair of regexps to email addresses
foreach (var pair in catchAllConfig.AddressMap)
{
	// Create the regular expression and the routing address from the dictionary.
	Regex emailPattern = new Regex(pair.Key);
	RoutingAddress emailAddress = pair.Value;
 
	// If the recipient address matches the regular expression.
	if (emailPattern.IsMatch(strRecipientAddress))
	{
		// And if the recipient is NOT in the address book.
		if ((this.addressBook != null) &&
			(this.addressBook.Find(rcptArgs.RecipientAddress) == null))
		{
			// Redirect the recipient to the other address.
			rcptArgs.RecipientAddress = emailAddress;
 
			// No further processing.
			return;
		}
	}
}

Note that in both cases as soon as we’ve got a positive result we stop further processing. If an address is in the ban list we immediately reject it, and if we find a matching regular expression we use its paired address to redirect the message and stop looking for further matches.

I’ve given enough information here that a C# developer with some spare time could create this Transport Agent themselves. However I know that it is really useful to have a package you can just download and install without having to worry about compiling it yourself. So I’ve packaged the DLL together with installation scripts and the install and configuration guide.

Please visit the product page for more information and to purchase the Transport Agent. Payment is via PayPal, you will receive the download link within minutes of payment.

It turns out that this is caused by the latest NVidia drivers – which of course I downloaded and installed on rebuilding the machine, the only fix at the moment is to add this to the registry:

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management]
"SessionImageSize"=dword:00000020

Hopefully NVidia will fix this in a future driver.

What do we buy the birthday boy who has a no-media policy?

That’s what Phil asked me before my birthday last month.  He was referring to the fact that I have no CDs, no DVDs no optical media of any description cluttering my living room.  All our TV and movies are stored on a variety of redundant storage and accessed over the network.  Be it full VOB rips of DVDs to Xvid or x264 encodes of movies and TV it can all now be played back by the new HTPC.  Of course with as much digital content as we have, we need a decent system to manage and search it.  Read on for the best solutions I’ve found…

MyMovies

This plugin provides a full management suite for keeping details of your movies.  What I particularly like is the standalone Collection Management tool that allows you to add and edit details from any machine on your network.  Once installed you can create categories and filters that, for example, limit the list on screen to just your HD movies.  The onscreen interface seems to be some kind of HTML variant rather than the “native” media centre applications, this is the only criticism I have of it.

Imagine having a version of IMDB tailored specifically to your movie collection.  That’s effectively what you end up with onscreen.  Just enjoyed a film and want to see another by the same director?  Click his name in the interface and you’ll find not only a biography but a list of all his other films that you have already got.  The same goes for the actors in the films.  You also get full cover art (front and back of the DVDs), audio track information and a brief synopsis of the film.

MyTV

The MyTV plugin reminds me of the Library functionality that I could never quite get to work in XBMC, only this time it works.  It queries thetvdb.com to find the titles, graphics, synopses and a myriad of other details for every episode of nearly every TV series you might have on your server.  Even if it can’t find them you can, in true collaborative style, add them to thetvdb yourself (as I have done for several shows).

Once it’s found the show, it relies on you having named the episodes correctly; for most TV downloaders the format will look familiar: Battlestar.Galactica.s01e01 would cause myTV to populate the database with the data from here.  Outstanding.

Both of these plugins work immediately out of the box and require relatively little faffing about.  They also both have very healthy communities built up around them offering support.  MyMovies has a contribution feature whereby if you find data missing in their database, you can upload if yourself earning points in the process.  You can also show off your entire collection via a free website should you want the world to see.

Hauppauge’s WinTV-NOVA-HD-S2 card is a Freesat compatible, HDTV capable satellite card. Hauppauge themselves provide (local mirror) a modification to the Windows driver that fools Vista into seeing a DVB-T card instead of DVB-S.  It’s a simple install: run the application and choose the satellite you intend to view – in our case Astra 2 at 28.2 degrees for the Freesat transponders – and it imports some settings into the registry to enable the device as a fake DVB-T card, and remaps the satellite transponders to appears as DVB-T channel muxes.

After a reboot you can start Vista Media Centre and perform a channel scan on your new tuner using the Sky headend (usually at the end of the list of available sources).  By default this will result in over 200 channels, most of which you can’t view because they’re protected by Sky’s encryption technology.

It’s worth editing the registry to remove transponders that you know you won’t need such as the multiple regional BBC and ITV channels, all the encrypted channels, and the myriad shopping TV channels.  Each transponder is stored as a key in the registry under:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HCW88bda\DVBS_as_DVBT

In my case, I used the LyngSat information on the Astra 2D satellite to compare the frequency and channel list.  The dvbs_freq value in the registry represents the frequency listed in the leftmost column on the LyngSat page. For example removing the registry entry with the value of 10788000 will get rid of a load of Midlands and Yorkshire BBC channels.  Just delete the entire key that contains the value and when Vista scans for channels it will skip over that transponder.

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HCW88bda\DVBS_as_DVBT\474000] ; #14
"dvbs_freq"=dword:A3B628 ; 10729000
"dvbs_polarity"=dword:2 ; 2
"dvbs_srate"=dword:55F0 ; 22000
"nit_onid"=dword:2 ; 2
"nit_tsid"=dword:7FA ; 2042
"nit_nid"=dword:0 ; 0
"DiSEqC_Port"=dword:1 ; 1

The only transponders I’ve left in are:

10729 (for E4, More4, Film4, Channel4)
10758 (for ITV2, CITV, ITV4, Men & Motors)
10773 (for CBBC, BBC3, BBC4, CBeebies, BBC2)
10818 (for BBC1 South)
10847 (for BBC HD)
10891 (for ITV1 South)

Until their contract expires with Sky, none of the Five network channels appear free to air.

Once you’ve scanned for channels you will end up with a somewhat unorthodox channel list.  You will need to rename some, and reorder and add channel listings to the rest.  You can do this via Settings->TV->Guide where you’ll find Change Channel Order, Edit Channels and Add Listings to Channel.  For example, by default Channel 4 appears as a channel named 5830.  You’ll need to rename this to Channel 4 and then add the guide listings for the appropriate channel.  Ordering channels is a real pain since you can’t just define channel numbers; you must labouriously move the channels up and down in the list.  Let’s hope that the Fuji update addresses this too.

Next post: the must have Vista Media Centre plugins for Movies and TV.

Out of the box the Vista Media Centre GUI is only capable of setting up 2 tuners at any one time.  This is odd since the underlying tuner architecture is actually quite capable of using as many tuners as you can fit in the machine.  Your limiting factor, really, is the speed you can push the data to your hard drive.

So, how to enable the extra tuners?  Here’s the process I followed to make it work:

1. Start Media Centre by pressing the Green Button.
2. Go to Setting -> TV -> Set Up TV Signal.
3. Follow the process for configuring the tuners, in the first run through you will configure the first 2 tuners.
4. Close the Media Centre interface.
5. Stop the Media Centre Receiver service.

   net stop ehrecvr

6. Open the Registry Editor, and navigate to this key:

   HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion
   \Media Center\Service\Video\Tuners\{71985F48-1CA1-11D3-9CC8-00C04F7971E0}

   That GUID on the end is the one for the DVB-T tuning space.

7. Under the GUID key there will be one key per tuner device – you need to find the two keys that were just configured by the Media Centre interface.
8. You can locate them by looking for the EnabledForMCE value under the UserSettings key.  If the value is set to 0xffffffff it has been configured, if not it’ll be 0×00000000.
9. Export those two keys to .reg files by right-clicking on the GUID key and choosing Export [Selected Branch Only].  Name each one separately (Tuner1.reg, Tuner2.reg, etc).
10. Repeat steps 1 to 9 for each set of two tuners you have, in my case I just had to do the process twice, once for tuners 1 and 2, then again for 3 and 4.
11. Once you’ve exported all the registry keys, and ensuring the ehrecvr service is still stopped, double-click all the keys to import them into the registry.
12. Close Registry Editor.
13. Start the Media Centre interface.

You can test that this has worked by entering the guide and starting recordings on 4 channels at once.  You should be able to go into Recorded TV and watch the streams of any of the channels as they are being recorded.

Sounds great, right?  So why am I using satellite cards instead of Freeview?  Two reasons: signal quality, and HD support.  We’re in a bit of a fringe signal area, and our aerial is fed via a series of boosters and distribution devices through multiple apartments.  I happen to have a Sky Minidish with a quad LNB anyway from when I used to subscribe to SkyHD.  I can’t put a new Freeview aerial on the roof, and loft-based aerials don’t offer the signal quality necessary.  So I turned to the satellite cards, which offer the benefit of HD content too.  More on them, and why using Satellite under Vista is a pain in the proverbial in the next post.

So, to disable this you must open regedit and find:

HKEY_CURRENT_USER\Software\Microsoft\Windows Desktop Search\DS

and set the ShowStartSearchBand value to 0.  That’ll give you back the default behaviour.

I find it quite irritating that the interface chooses to admonish me that I’m not indexing the content instead of just presenting me with the search functionality that is able to search the content for me.

In order to play these files in VMC I had to install a few extra codecs and applications:

Haali Media Splitter

This is the standard app for any system trying to play video stored in the Matroska container format. It “splits” the file, piping the video, audio and – if present – subtitles to separate codecs for further processing. It does not do any decoding itself, merely allows other decoding apparatus to see the streams. It also allows the system to see into the containers to create thumbnails – which makes things in VMC much prettier.

ffdshow tryout

This does nearly every other codec you can imagine. During installation on the mixer settings screen I turned the mixer off. After installation I ran the audio and video configurations to make a few more changes: AC3 and DTS audio were set to spdif: this allows ffdshow to direct the audio stream directly to the optical out for my A/V Receiver to process.

Also, in the Video Configuration I set x264 decoding to disabled, because instead of using ffdshow, I used:

CoreAVC

CoreAVC (which actually includes Haali as part of its setup) is a multi-core aware AVC video decoder. With a fast enough processor it is capable of decoding 1080p MKV files while ffdshow pushes DTS audio to my receiver. It’s not free – setting you back a measly 15$ – but is well worth the price. Rumour has it that ffdshow will soon be multi-core aware, if so it would mean you could do without CoreAVC. At the time of my build however, this seemed the best solution.

In the CoreCodec Settings, make sure that Explorer Integration is set to Yes – this enables thumbnailing.

In order for VMC to recognise .mkv files as video, I had to add these settings to the registry:

Windows Registry Editor Version 5.00
[HKEY_CLASSES_ROOT\.mkv]
@="CoreMedia.MKV"
"PerceivedType"="video"
 
[HKEY_CLASSES_ROOT\.ts]
@="MatroskaVideo"
"PerceivedType"="video"
"Content Type"="video/x-matroska"

Save that as “mkv.reg” and double-click to add it to the registry.

With those applications installed and with those few changes made your system (assuming decent processor) will be able to play 1080p x264 mkv files, complete with 5.1 DTS audio. You may congratulate yourself with a cup of tea. Come back soon for why I’m using satellite TV cards rather than Freeview (and why doing so is a pain in the proverbial in the current version of VMC) also: the must have VMC plugins to manage your movie collection and TV episodes.

All I wanted to do was right-click a selection of mp3s and then queue them in the Winamp playlist.  That’s when I noticed a load of entries like “sync to device”, “burn”, “add to library”.  Turns out Media Player had hooked into the shell.  Thankfully, there is a way to get rid of this integration (which added a delay of several seconds to my right-click!).

Here’s the fix, click Start, Run and type:

regsvr32 /u wmpshell.dll

Horrah – a much cleaner context menu.  I wonder if there is any software to allow you to remove specific entries from that list?

That particular graph shows the usage of my ADSL connection over time; green is downloaded data, while the blue line shows my uploads. It is generated by Tobi Oetiker’s outstanding MRTG – The Multi Router Traffic Grapher. MRTG is a free Perl script that connects to any device capable of talking SNMP and pulls interesting data from it.

I currently log DSL, Wireless and LAN use (Bytes per second), along with line noise and Noise Margin of the DSL connection. I could also log what the switch in my network is doing, and even what my various servers are up to. As you can see from the graphs you get a nice indication of traffic patterns over time – it’s very easy to spot peaks when BitTorrent kicks in and out, or when someone in a different time zone is playing on my Counter Strike server!

It gives me an “at a glance” overview of what most of my networked devices are doing – which was enough to make me notice the other day that despite having all my wireless devices switched off, something was still polling the Access Point. Turns out I’d left one of the laptops disassociated – but on – and it was sending a little chatter to the AP every few hours.

• Use Disk Utility (on the G5) to create a DMG file.
• Connect the MacBook Pro and G5 to my gigabit network, and boot the MacBook Pro into Tiger.
• Connect the USB drive (a self powered 100Gb 2.5″ drive in this case) to the MacBook Pro (I used the right USB port – not sure if it matters which).
• I erased the (NTFS) partition on the external disk, and created a GUID Apple partition.
• Select the Restore tab on Disk Utility and drag the DMG file to the source field.
• Drag the USB partition (the one I just created) to the destination field.
• Click Restore.
• After it’s finished, you should be able to open System Preferences and find the USB disk in the Startup Disk pane.
• Select it and press restart – the MacBook Pro reboots and boots off the external disk.
• From here on it’s as if you were using the DVD!

The install was pretty swift – without the extra printer drivers, fonts or X11 it took about 10 minutes to install. Since installing Leopard has killed rEFIt the next step is to reinstall Windows and add the Boot Camp 2.0 drivers.

Oh, and the Leopard intro movie is quite pretty, but has really silly music.