This is one of the most widely used integration patterns and also the most asked interview question.
In this pattern a huge message (also called batch file) is debatched and sent to multiple or single recipient which process these messages and sends a response or acknowledgement back. Later all these responses are gathered and collated into a single message in Parent Orchestration. So basically this complete process can be divided into two sub processes- Splitter Pattern and Gatherer pattern.
You can read more about the Scatter and Gather Pattern here.
There can be many ways to implement this pattern in BizTalk but below implementation is asynchronous and loosely coupled.
Basics:-
a) For debatching the huge message (batch file) I have called/executed Receive Pipeline with XML Disassembler component from within the Parent Orchestration. This acts as Splitter.
b) Sent out the debatched messages using Start Orchestration shape, this will allow asynchronous processing of all the child messages(debatched messages)
c) For gathering the responses in parent orchestration I have used Self Correlation, acts as Gatherer
Detailed Steps:-
Parent Orchestration
1) Create a parent orchestration to receive input message (batch file) with “Transaction Type = Long Running”. This scope should be long running because it will contain an Atomic Scope for debatching.
2) Do some processing/transformation if required.
3) Create an Atomic Scope for debatching the big message. Here the scope should be Atomic because we will be calling/executing Pipeline within it and the pipeline is of non- serializable type.
4) Execute Receive Pipeline (having XML Disassembler component) in Expression Shape for debatching.
varReceivePipelineOutput= Microsoft.XLANGs.Pipeline.XLANGPipelineManager.ExecuteReceivePipeline(typeof(TestDebatchedProcessing.ReceivePipeline_DebatchInput),msgFormatted);
Here “varReceivePipelineOutput” is a variable of type- “Microsoft.XLANGs.Pipeline.ReceivePipelineOutputMessages” and should be defined within the Atomic Scope created for debatching. This class extends IEnumerator interface so implements GetCurrent and MoveNext methods.
Note: –
You will have to add reference to “Microsoft.XLANGs.Pipeline” dll present at below location – <Install dir>Microsoft BizTalk Server 2013Microsoft.XLANGs.Pipeline.dll
Read more about debatching XML messages in orchestration here.
5) Loop through all the debatched messages using “varReceivePipelineOutput.MoveNext()”. MoveNext function indicates whether the pipeline should move to the next message.
6) Construct Child Message using Message Assignment shape.
msgChild=null; varReceivePipelineOutput.GetCurrent(msgChild); ->This will retrieves the current XLANG message.
7) Increase the count variable.
intCount = intCount + 1;
8) Start Child Orchestration for further processing by passing Child Message and PortType (Created to receive response from Child Orchestration at step 10).
Note: –
Steps :-
a) Right Click on the Port Surface -> New configured Port- > Give PortName and PortType
b) Select Port Direction as “I’ll always be receiving message on this port” and Port Binding as “Direct” and type as “Self Correlating” as shown below.
9) Create a Loop to receive all the messages sent out. Here we loop till the value of intcount becomes zero.
intCount > 0
10) Receive Processed message from the Child Orchestration, this receive shape is connected to the Self Correlated Receive port created in step 8-b-2
11) Reduce count variable
intCount= intCount -1;
Child Orchestration
1) Create a Child Orchestration which receives debatched messages from Parent Orchestration and processes it further.
As first step create Orchestration Parameters
a) Port_SelfCorr -> Port Parameter to receive PortType
b) msgChild
3. Do some processing (I have done some transformation).
4. Send the Processed Message back to Parent Orchestration. This send port is connected to the port created while creating Port Parameter
You can find the complete solution here.
Contact Me:-
@Gmail, @Facebook , @Twitter, @LinkedIn , @MSDNTechnet, @My Personal Blog
Advertisements
My First Post
It’s been awhile since I told Mike that I’ll be putting up a post.I hesitate to write after reading all the deep technical contents Mike has been writing.I have always envisioned my role to consist of random ranting of daily routines in BizTalk CPR.I will work towards that but one has to start somewhere.
Thoughts on Orchestration Performance
I’ve promised Mandi a KB on Orchestration Performance for awhile now (Sorry Mandi).The draft was done until my HD crashed without a valid backup.Good thing that I have the main points imprinted into my brain.I will post the ideas here first so I can get some feedbacks first regarding to the validity of my memory imprint.
First, why do we want to talk about Orchestration performance?From support standpoint, we get a lot of cases on performance.Fine tuning performance is difficult and time consuming once a solution is in production.Also, no matter how experienced is the support engineer, he/she will not know your design better than you.The best thing for a BizTalk developer/administrator is to address these common issues before going live with any solution.
Emphasis on Test
Given the complexity of Orchestration designs and the permutation of hardware components, it is nearly impossible for anyone to predict performance.We do get cases from time to time from customers who want to hear our best effort estimate.The keyword here is ‘estimate’.Only way to find out for certain how much load your hardware can handle and whether your design has potential bottlenecks is by running proper stress tests.Both the Microsoft BizTalk Server Operations Guide and the upcoming BizTalk Performance Optimization Guide outline the proper testing methodology.
Host Level Isolation
It is a good idea to isolate your Orchestrations in a separate host.Not only does the separate host instance provide you with dedicated process level resources for your Orchestrations, it also allows more specific fine tuning of host level throttling configuration.If your Orchestration host is experiencing high memory utilization, you can consider separating your Orchestrations further into more hosts.Of course, you should do this with understanding of your hardware capacity.No point adding more host instances if you are reaching the limit of what your hardware can handle.
There are many ways to organize your BizTalk artifacts with regard to hosts.You can certainly do it by functionality or operation so if you have to temporarily bring down a host, it’ll affect only a single functionality, not all of your workflows.Also, if you know ahead of the time that two Orchestrations are designed to be running concurrently, they can benefit from being in different hosts instead of competing for resources in same host instance.
Throttling
Whenever you feel your Orchestration processing is slowing down, you should first take a look at the Performance Counters to see if throttling is taking place.The counters you want to keep track are under the BizTalk: Messaging Agent performance object.If publishing or delivering throttling is taking place for a host, you will get a non-zero status for the respective counter.You can then look up the status code in BizTalk Documentation.Please consult Host Throttling Performance Counters for more information on these counters.
Please use BizTalk Documentation (How to Modify Default Host Throttling Settings) as a guide when you modify throttling behavior.Process Memory Usage may be one setting you want to consider changing when you go from development to test/QA or production.By default Process Memory Usage is set to 25%.When the address space of a host process is 2GB, throttling may kick in after process memory usage crosses over 500MB, which may be rare in development but not during production.For a 32bit host, do not configure your host to use more than 1.54GB of memory (75%).If you have a 64bit host, you can change the setting to 100% so BizTalk host will not throttling until it uses more than 2GB of memory.
One thing to remember is that your Orchestration host may not be the one that is throttling but Orchestration performance is still affected.For example, your Orchestration may be bound to a SQL send port.If the host that is running the SQL send adapter is under throttling, messages cannot be published and your Orchestration may be held up as result.As a general rule, if you have throttling, you should investigate the cause and determine if you can do something about it.
Messages/Variables scope
We often see cases where Orchestration instances use a large amount of memory.One thing you should keep watch during development is the scope of your messages and variables.Any message or variable can either be defined at the Orchestration level or bound to a scope.It is similar to global variable versus local variable in any programming language.While Orchestration engine does have logic to invoke garbage collection on messages that are not used later on in the Orchestration, you don’t have absolute control over when GC kicks in.With large messages or object types, it makes sense to define them in the scope they will be used.Furthermore, if you load large objects into variables, such as a large XML into XMLDocument, it is good practice to clean up as soon as you are done with it.
When you are running your tests, keep an eye on the memory footprint.These issues should be resolved during design or test stage.Once you are in production, there is no easy way for us to know what is loaded in memory, without getting a memory dump.That is time consuming and you’ll end up with downtime while someone gathers and analyzes the memory dump.
More information on how to troubleshooting out of memory exception with BizTalk Server:
http://support.microsoft.com/kb/918643
Persistence Points
For longer running Orchestrations to maintain state information, the Orchestration engine saves instance state to the BizTalkMsgBoxDB at various persistence points.
Guaranteed persistence points are: Washington state id formula.
·Send Shape (after a message is sent)
·Start Orchestration Shape
·Suspend Shape
·End of a Transactional Scope (atomic or long-running)
·An Orchestration Debugger breakpoint is hit
·Orchestration Engine determines that the instance needs to be dehydrated
·When Orchestration Engine is shutdown, whether through controlled shutdown of the host or abnormal circumstances.In this case, the engine tries to persist but if that fails, Orchestration instance will resume from last successful persistence point.
Receive Shape, Listen Branch, and Delay Shape are conditional persistence points.Receive Shape dehydration behavior is determined by the history of how long Orchestration Engine waited for this subscription and the threshold value configured in BTSNTSVC.EXE.config file.Listen Branch and Delay Shape dehydration behaviors are based on the configured timer value and the threshold value.Please reference “How to Calculate Dehydration” in BizTalk documentation for more details.
Since each persistence point requires your Orchestration instance and the associated data to be serialized to the database, it can impact performance if you have large number of persistence points or there is large amount of data to be serialized.During design time, you can use the above list of persistence points as a guide to adjust your workflow.During runtime, you can use the dehydration and persistence counters under XLANG/s Orchestration performance object to determine whether excessive persistence may be a factor in your performance issue.
To prevent persistence, you can enclose a potential persistence point within an Atomic Scope.Orchestration Engine won’t persist until it reaches the end of the transaction.Be careful when you do this.While it does keep the Orchestration instance from persisting, it also forces the Orchestration instance to remain in memory when it could have dehydrated.You may run out of system resources if you have a large number of active Orchestration instances.
How To Use Start Orchestration Biztalk Video
Map
Biztalk Singleton Orchestration
Transformations are more efficient when they are executed from send or receive port.There are scenarios where it makes sense to have a map in the Orchestration.For example, you can execute a transformation dynamically within an Expression Shape and also maps in Orchestration can accept multiple inputs.In general, unless you have an absolute requirement to use map in Orchestration, you should avoid during so.
Distinguished Field/Promoted Property Field
When you promote a property, consider how you’ll be using it.If you plan to only access the data within an Orchestration, promote the property as Distinguished Field only.Promoted Property Fields can be used for routing and they are more expensive. It is not necessary unless you plan to use it outside of Orchestration, for example, as filter for a Send Port subscription.
External Components
Many people design Orchestrations that use external assemblies within Expression Shape.This gives your workflow more flexibility.However, when you call a method from external assembly or unmanaged code, you should realize that the Orchestration Engine has little control over what happens within external code.If the code fails, your Orchestration design needs to handle the exception.If the method call does not return in a timely manner, Orchestration Engine will have to wait on it.You should not count on the Orchestration Engine to force the termination of external component through any timeout mechanism.If a timeout mechanism is desired, you should build it into the method you are calling.
With any custom code, you should implement tracing or debugging information.I have seen many cases where Orchestration instance fails within custom code.However, since there was no tracing within the component, we spent time on isolating the failure.Once the problem was isolated to the custom component, the issue still needed to be resolved by the original developer since we didn’t have any knowledge of the component.It is more efficient for you if we can quickly determine what are failing and who should be engaged to fix the issue.
Latency
You may experience slow Orchestration performance when first message comes in after long idle.This may be caused by application domain unloading within the BizTalk host process.If an AppDomain is shutdown after idle, the next message that comes needs to wait for the Orchestration to compile again.Depending on the complexity of your design, this can be a noticeable wait.To prevent this in low latency requirement scenario, you can modify the BTSNTSVC.EXE.config file and set SecondsIdleBeforeShutdown property to -1.This will prevent AppDomain shutdown due to idle.
Tracking – Orchestration Events
Document tracking can impact performance throughout BizTalk, not just Orchestration.For Orchestration, you should realize that Orchestration Event Tracking is on by default.This is useful during development and testing since Orchestration Events are required for Orchestration Debugger.However, if you do not intend to debug an Orchestration directly in production, you should turn Orchestration Event Tracking off.Orchestration Events are eventually moved to DTA_DEBUGTRACE table in BizTalkDTADB by TDDS.We have seen slow read/writes to this table once it gets large (several hundred thousand rows).What is considered large may vary due to your SQL Server hardware.In the end, if TDDS cannot move data efficiently into BizTalkDTADB, data is accumulated in BizTalkMsgBoxDB.Large MsgBoxDB can cause all of your hosts to slow down, and eventually lead to throttling.
If you must track Orchestration progress, it is better to implement a BAM solution.It has lesser performance impact.
Thread Pool Size
Orchestration Engine and many BizTalk Adapters use worker threads from managed thread pool.The default size for the thread pool, which is defined by MaxWorkerThreads in registry, is 25.As Orchestration instances are dehydrated, threads are returned to the thread pool.While the default setting may be adequate for many scenarios, if you have large number of Orchestration instances that need to be active concurrently, you may be limited by the thread pool size.You can usually detect this by using the Running Orchestrations performance counter under XLANG/s Orchestrations performance object.You may notice that the number of running Orchestrations increases and then remains constant once it hits the thread pool limit.Of course, before you suspect thread pool is the limiting factor, you should first make sure no throttling is taking place and your system is not already under stress.
You can increase the thread pool size by modifying the MaxWorkerThreads value in registry.For instruction on how to do this, please reference http://support.microsoft.com/kb/900455.
Orchestration Profiler
It is often useful to find out which shape in your Orchestration is taking up most of the time.With that information in hand, you can then proceed to troubleshoot the bottleneck.Orchestration Profiler (http://www.codeplex.com/BiztalkOrcProfiler) is a nice tool that shows you the max and average duration spent on each shape.In addition, it displays your test coverage to let you know if you’ve covered all branches of the workflow.It is not an official tool so you may want to take that into consideration if you want to run it in production.But it is certainly a useful testing tool.
For more information about Orchestration in general, please reference the Orchestration FAQ:
More to come….
This article would help you visualize the key development design in BizTalk. The main process which invoke child process by Call and Start orchestration shapes.
Lets develop a 1st child orchestration which would be invoked by the main process using call orchestration shape. Remember, call orchestration returns the control back to the caller orchestration its a synchronous process, (happens in same thread).
Add a new orchestration and in the orchestration view to add parameters right click and choose appropriate type, lets add message and variable parameters with ref and out as its directions respectively. Then we would add a simple expression shape and do the following manipulation.
Now lets, build our 2nd orchestration which would invoked using start orchestration shape. This would be asynchronous process. Create a new orchestration and add the message parameter to it with direction as In. This would assign a copy of the in message to a new message and publish it to destination system.
With the child process ready, now we will build the main process as below, drag receive, call start, message assignment and send shapes into the orchestration canvas as shown below.
In order to configure Call and Start Orchestration shapes, double click on the shape and choose the appropriate child process and provide the parameters
Transformation can be achieved using an assignment shape by doing a dynamic transformation.
The Orchestration can not promote the properties to message context by default like pipelines can do by using the msg.Context.Promote() method. Inside Orchestration one has to create a correlation set and intialize that correlation set while publishing the message. Then the BizTalk Engine will promote the property to the message context.
Any exception which is not caught within the exception handlers of the orchestration causes the orchestration instance to be moved to the Suspended state. By applying the appropriate error handling in an orchestration it can be prevented entering the Suspended state.
One of the best practices when implementing orchestrations is to use orchestration patterns when possible. These patterns are basically design pattern, which is a general reusable solution to a commonly occurring problem within a given context of BizTalk orchestration. This article will provide some useful resource links to aid you in using patterns when implementing an orchestration.
Correlation is the process of matching an incoming message with the appropriate instance of an Orchestration.
A correlation type is a list of properties that eventually populates with values for use in routing messages.
The term binding refers to the configuration of orchestration ports in order to control the creation of subscriptions and/or promoted properties. Binding is used to control how messages will be routed to or from orchestration ports by the subscription mechanism.
Calling an orchestration will use the same thread to run another orchestration while using Start Orchestration will create a new thread to run the started orchestration.
A Call Orchestration returns the control back to the caller. A Start Orchestration shape starts the orchestration in a non-deterministic way. As a conclusion, Calling an Orchestration will be synchronous operation where the caller waits for a response, while Start Orchestration is asynchronous operation.
Pro's:
☛ Loose coupling ☛ Not bounded to any physical port (explicitly) Con's: ☛ Can lead to subscribing to messages published by another publisher
Promoted properties can be used as criteria in message routing while written properties cannot. Property field is a Promoted Property in the context. Distinguished field is a Written property in the context.
The engine will save the state of an orchestration in the following circumstances:
☛ Send Shape (after a message is sent) ☛ Start Orchestration Shape ☛ Suspend Shape ☛ End of a Transactional Scope (atomic or long-running) ☛ An Orchestration Debugger breakpoint is hit ☛ Orchestration Engine determines that the instance needs to be dehydrated ☛ When the Orchestration Engine is shut down; through the controlled shutdown of the host or abnormal circumstances. The engine tries to persist but if that fails, the Orchestration instance will resume from the last successful persistence point. Comments are closed.
|
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |