carbon14 Blog

Recently I ran into an unexpected problem with a custom pipeline component which I thought had been working fine. Suddenly it stopped working, but with an unexpected workaround. Today I worked out what the problem is, and although it makes perfect sense it caught me by surprise. I can't find any mention of this gotcha elsewhere on the internet, so I thought I'd write about it here.

Last year I wrote a custom pipeline component (Message Dump component) which optionally dumps the message body and/or the context of the message, and I use it regularly in my pipelines. Because the dumping is optional, and can be quickly controlled at runtime, I place the component in pretty much all of the pipelines that I use. When turned off, the overhead is minimal, but when you want to check what's going through any given stage of a pipeline you can quickly turn it on and get what you need.

This year we decided that one of our pipelines should always dump a copy of the message body and so I created another cut down version of this component (Archive component) that didn't have the optional switch. It has just 1 configuration option which is the path to the file that should be written, and that file path can contain various macros which can be expanded from the message context.

Everything seemed to work in the development and test environments; we went live with it and it was great. After a period of live commissioning I turned off the Message Dump components in the pipeline, and that's when it all went wrong.

[More:]

This is my pipeline:

    Decode Stage:
        MessageDump component 1
    Disassemble Stage:
        XML disassembler component
    Validate Stage:
        MessageDump component 2
        Archive component
    ResolveParty Stage:
        Party Resolution component
        MessageDump component 3

Suddenly the archive component was archiving the message in the wrong place. The macro in the file path was not being replaced, so every message was being written into the same file, rather than each message getting it's own file.

But if I turned back on dumping the message prior to the Archive component then it was working fine. (MessageDump component 2)

The macro in the path that was not being replaced, was a context property being promoted from the message by the XML Disassembler component. This didn't seem to make sense. If it wasn't being mapped into the file name, that probably meant that the context property didn't exist, yet if I turned on the Message Dump component which wasn't attempting to change the message at all, then suddenly it would all work again. If I let the message suspend in the message box, I could check it's context from within the Group Hub, and the required property was there exactly where I expected it.

Attaching the Visual Studio debugger to BizTalk I was able to see what was going on, but it was still confusing. When my Archive component was executed, the context did not contain the property I was after, which should have been promoted by the XML disassembler in the previous stage. But if I enabled the MessageDump component 2 to do its thing, then the property would show up. Somehow the MessageDump component seemed to be influencing the success of a previous component.

This is what I figured out is going on:

When a pipeline component is executed, it receives a stream containing the message. It can ignore this and pass the stream on to the next stage untouched. It can pass a new stream to the next component and perform some sort of conversion; reading from the input stream and writing to the new stream it has passed on. Or in the case of my custom components it can read the message, reset the stream and pass it back out. Not every stream implementation that you might receive can be rewound so Microsoft have helpfully provided an implementation of a wrapper that you can use to turn a forward-only stream into a seekable stream.

Think about what that all means. BizTalk executes the pipeline components one at a time, in order. At the end of the process there is a stream which BizTalk can read to get the message from the bottom of the pipeline. But if there are no components in the pipeline, the stream at the bottom is the same as the stream at the top, untouched and unread. Even if there are components in the pipeline, they might not touch the stream either. Although they've been executed, the stream has never been read...

My problem occurs because of what the XML Disassembler does. When executed, the XML Disassembler component unpacks the content of any envelope in the message and then wraps a new stream around that content to pass on down the pipeline, but crucially it does not read that message. Execution of the stage finishes and BizTalk executes the components in the next stage. Here's the punchline, the XML disassembler will promote properties that it recognises in the message to the context of that message, but only when it reads the property; and that doesn't happen during the execution of the pipeline stage, it happens somewhere in the read method of the stream that was passed on down the pipeline. For a component which reads the context during execution, such promotion hasn't necessarily happened yet.

With my MessageDump component turned off, I was received the stream from the XML disassembler component containing a message as yet unread. I also received the message context which did not yet have the promoted property, so my file path macro was not expanding and I was seeing my problem.

With the MessageDump component turned on, that component was reading the whole message in order to dump it, and in the processes, the XML disassembler stream was reading the message and finding properties to promote. By the time my Archived component got hold of the message stream, the property had been added to the context and everything worked.

I guess the take home from this is to be aware that some things that happen in the pipeline might be side-effects of actually processing the message, and there's little guarantee that message processing occurs at the same time as component execution

Take Care.