Sender: MQSeries List <MQSERIES-0lvw86wZMd9k/bWDasg6f+***@public.gmane.org>
In-Reply-To: <9BB766F505133C498D6D60FB74AA403C2E680304-Sa/7lAhLIlFOSYu4ox5p5Tfn7f/***@public.gmane.org>
<mailto:LISTSERV-0lvw86wZMd9k/bWDasg6f+***@public.gmane.org?body=INFO%20MQSERIES>
List-Unsubscribe: <mailto:MQSERIES-unsubscribe-request-0lvw86wZMd9k/bWDasg6f+***@public.gmane.org>
List-Subscribe: <mailto:MQSERIES-subscribe-request-0lvw86wZMd9k/bWDasg6f+***@public.gmane.org>
Personally I would prefer to put the dispatcher in anyway. 12 different apps,
enough time and money on testing right?!), all reading from the same queue ...
performance suddenly declines. :)
Just my two penn'orth.
Darren.
Post by Potkay, Peter M (CTO Architecture + Engineering)In other words, if you go the dispatcher route put an MQ queue in front of
each of the 12, and put the dispatcher in between the original input queue and
the 12 new queues.
But first do some testing to see if this extra complexity is required. It’s
possible all 12 of your apps running in parallel on the queue are going to be
able to keep the queue depth at or near zero and this whole conversation is a
moot point.
*Peter Potkay *
*Potkay, Peter M (CTO Architecture + Engineering)
*Sent:* Sunday, September 21, 2014 9:38 AM
*Subject:* Re: Message selectors
“My idea is to have a single application read messages from the queue with no
selectors, examine the property value, and pass the message directly to the
appropriate backend application.”
A dispatcher.
If the volumes were going to be high and the distribution of 12 wildly uneven
AND the consumers were going to be slow meaning the single input queue would
frequently have a significant queue depth, this is a very good idea. As long
as the dispatcher can quickly hand off messages and go for the next message.
But if the dispatcher gets stuck twiddling its thumbs waiting for backend app
#10 to finish its previous transaction before it can hand #10 the next
message, while other messages pile up, this dispatcher model will actually
make overall processing much, much worse. Without the dispatcher at least all
12 backend apps can have concurrent handle on the input queue and the queue
manager can be servicing all 12 at once. Granted there may be some higher CPU,
and the lonely #12 app that gets one message a day compared to #1 that deals
with thousands a minute might find its MQGET call taking a few tenths (?) of a
second longer, but overall it would probably work better than the dispatcher.
Who cares if you drive the CPU hard – that’s what it’s there for. But if the
dispatcher can hand off the messages quickly and not find itself waiting for
one of the 12, maybe even have the power to spawn additional threads of the
back end consumers when needed, then go that route.
*Peter Potkay *
*Sent:* Sunday, September 21, 2014 12:07 AM
*Subject:* Re: Message selectors
Ah, I see...
Perhaps I should explain further -- the incoming messages all contain a
property that has one of 12 values. There are 12 backend applications, each
designed to handle just one of these property values. The architect wants each
of these 12 applications to open the queue in shared mode and issue an MQGET
with the selector value it's seeking. Since the property values are not evenly
distributed, it's possible one of the modules will be forcing a sequential
read through the queue for many hundreds (if not thousands) of messages.
My idea is to have a single application read messages from the queue with no
selectors, examine the property value, and pass the message directly to the
appropriate backend application.
Does that clarify the situation?
Thanks,
RW
Are you implying that "modern" queue managers somehow index ALL message
properties?
No. I’m implying that having the QMgr select the messages locally and then
send the only the relevant ones over the wire is a **lot** faster than
sending every message over the wire, evaluating the selectors in the
client libs, then having the client libs return to the calling program
when they find a message of interest. If the selection returns every
message it won’t matter. But if the selection returns a subset, the
performance increase is proportional to the inverse of the size of the
selected subset. The smaller the selected subset, the greater the
difference in performance between QMgr vs client-side selection.
-- T.Rob
Behalf Of *RW
*Sent:* Saturday, September 20, 2014 11:44 AM
*Subject:* Re: Message selectors
T.Rob, you wrote, "getting high performance requires that you use a modern
client and QMgr which allows the QMgr to perform the selection."
Yet my understanding is that there's only a subset of fields in the MQMD
that are indexed, thereby allowing for fast selection. Are you implying
that "modern" queue managers somehow index ALL message properties?
Thanks,
RW
This is not exactly an apples-to-apples comparison. Such a front-end
program would need to know the modules and selection criteria ahead of time.
That implies static selection and processing the messages in real time, with
the ability to restart and process messages collected in a queue as an
exception case.
On the other hand, use of selectors on a deep queue assumes something reads
them *after* they were produced. This implies both dynamic selection and
that reading stored messages is routine rather than the exception.
As others have noted, WMQ selects on things in the message envelope whereas
a custom program or Broker selects based on things in the payload. When you
can use WMQ's selection, getting high performance requires that you use a
modern client and QMgr which allows the QMgr to perform the selection. In
older versions (and, I believe in Compatibility Mode) the client reads all
the messages, evaluates the selectors, then delivers to the calling program
the msgs that meet the selection criteria.
The "right" answer is the one that meets the actual business requirements
and those were not provided with the question so it's hard to provide useful
feedback at this point. The one thing it is easy to recommend is that
whatever you do, make sure that the thing evaluating the selector, whether
that's the QMgr, a Broker instance, or your custom code, runs local on the
same host as the QMgr and uses a Bindings Mode (in memory) connection.
Kind regards,
-- T.Rob
T.Robert Wyatt, Managing partner
IoPT Consulting, LLC
+1 704-443-TROB
https://ioptconsulting.com
https://twitter.com/tdotrob
-----Original Message-----
Of RW
Sent: Friday, September 19, 2014 14:12 PM
Subject: Message selectors
What's the impact of using message selectors on a queue with a lot of
messages? Surely it would degrade performance since the QM has to
sequentially search the queue?
I'm having a discussion with an architect who wants to use message
selectors in order to have a series of application modules that only
retrieve relevant messages for processing. I'm suggesting a front-end
application that reads messages from the queue as they arrive, with no
selectors, which then has logic to choose the application module to invoke
based upon examining the message properties after the message has been
read.
Thoughts? Experiences?
Thanks,
RW
message body (not the subject), write: SIGNOFF MQSERIES Instructions for
managing your mailing list subscription are provided in the Listserv
General Users Guide available athttp://www.lsoft.com
Archive:http://listserv.meduniwien.ac.at/archives/mqser-l.html
in the message body (not the subject), write: SIGNOFF MQSERIES
Instructions for managing your mailing list subscription are provided in
the Listserv General Users Guide available athttp://www.lsoft.com
Archive:http://listserv.meduniwien.ac.at/archives/mqser-l.html
____________________________________________________________
Low-Cost Flood Insurance
Find a policy in your area and get a free flood risk profile!
http://thirdpartyoffers.netzero.net/TGL3265/541ca3c6708c723c67e0amp05duc
--------------------------------------------------------------------------------
List Archive <http://listserv.meduniwien.ac.at/archives/mqser-l.html> -
Manage Your List Settings
<http://listserv.meduniwien.ac.at/cgi-bin/wa?SUBED1=mqser-l&A=1> -
Unsubscribe
Instructions for managing your mailing list subscription are provided in
the Listserv General Users Guide available at http://www.lsoft.com
<http://www.lsoft.com/resources/manuals.asp>
--------------------------------------------------------------------------------
List Archive <http://listserv.meduniwien.ac.at/archives/mqser-l.html> -
Manage Your List Settings
<http://listserv.meduniwien.ac.at/cgi-bin/wa?SUBED1=mqser-l&A=1> -
Unsubscribe
Instructions for managing your mailing list subscription are provided in
the Listserv General Users Guide available at http://www.lsoft.com
<http://www.lsoft.com/resources/manuals.asp>
____________________________________________________________
<http://thirdpartyoffers.netzero.net/TGL3256/?u=http://newsletter.adsonar.com/nwrss/iMapRedirector?placementId=1560786&plid=379054&pid=2070767&ps=36140222&rotation=4&type=2&pos=0&zw=500&zh=70&v=5&url=NA&uid=>Image
removed by sender.
--------------------------------------------------------------------------------
List Archive <http://listserv.meduniwien.ac.at/archives/mqser-l.html> - Manage
Your List Settings
<http://listserv.meduniwien.ac.at/cgi-bin/wa?SUBED1=mqser-l&A=1> - Unsubscribe
Instructions for managing your mailing list subscription are provided in the
Listserv General Users Guide available at http://www.lsoft.com
<http://www.lsoft.com/resources/manuals.asp>
************************************************************
This communication, including attachments, is for the exclusive use of
addressee and may contain proprietary, confidential and/or privileged
information. If you are not the intended recipient, any use, copying,
disclosure, dissemination or distribution is strictly prohibited. If you are
not the intended recipient, please notify the sender immediately by return
e-mail, delete this communication and destroy all copies.
************************************************************
--------------------------------------------------------------------------------
List Archive <http://listserv.meduniwien.ac.at/archives/mqser-l.html> - Manage
Your List Settings
<http://listserv.meduniwien.ac.at/cgi-bin/wa?SUBED1=mqser-l&A=1> - Unsubscribe
Instructions for managing your mailing list subscription are provided in the
Listserv General Users Guide available at http://www.lsoft.com
<http://www.lsoft.com/resources/manuals.asp>
************************************************************
This communication, including attachments, is for the exclusive use of
addressee and may contain proprietary, confidential and/or privileged
information. If you are not the intended recipient, any use, copying,
disclosure, dissemination or distribution is strictly prohibited. If you are
not the intended recipient, please notify the sender immediately by return
e-mail, delete this communication and destroy all copies.
************************************************************
--------------------------------------------------------------------------------
List Archive <http://listserv.meduniwien.ac.at/archives/mqser-l.html> - Manage
Your List Settings
<http://listserv.meduniwien.ac.at/cgi-bin/wa?SUBED1=mqser-l&A=1> - Unsubscribe
Instructions for managing your mailing list subscription are provided in the
Listserv General Users Guide available at http://www.lsoft.com
<http://www.lsoft.com/resources/manuals.asp>
To unsubscribe, write to LISTSERV-0lvw86wZMd9k/bWDasg6f+***@public.gmane.org and,
