DataPower is an appliance. I won't be able to run that code.
But I do have a PMR open asking for confirmation that by default DP spits out little endian data.
From: MQSeries List [mailto:***@LISTSERV.MEDUNIWIEN.AC.AT] On Behalf Of Tim Zielke
Some of you are probably already aware, but if you wanted to determine programmatically the endianness of your server, you could do something like the following C program.
printf("little endian. crack egg on small end.\n");
printf("big endian. crack egg on big end.\n");
Peter - It might be a good confirmation to run that program on your DataPower server with the MQ Client. It should come back little endian based on your analysis. If it comes back big endian, something else is going on.
From: MQSeries List [mailto:***@LISTSERV.MEDUNIWIEN.AC.AT] On Behalf Of Darren Douch
A very good piece of detective work in the end Peter, but I would take a different view of the lesson to be learned, especially when the developers sit down in class next time. ;)
The lesson is to first understand your data and only rely on the default ccsid/encoding values once you've made a conscious decision that they accurately describe it.
Post by Potkay, Peter M (CTO Architecture + Engineering)Here is the end result for this problem and solution.
Messages sent from Datapower via MQ to an OTMA Bridge queue on the
mainframe were causing the IMS transaction to abend.
*Solution: *
By populating the MQMD.Encoding field in the MQ message to 273 the
abends stopped and a good reply came back from IMS. We figured this
out last Friday, the question was why did this work. Was it a proper
solution, or a hack to get around a bug and the hack solution might
itself break in the future? We now know this is the right solution.
1.WTX Developer A was using a WTX Map to create the application
specific message payload in the MQ message. He was referring to how
another app first did it in their Java code years ago to understand how to lay the data out.
2.Datapower Developer B was creating the MQMD header and the MQIIH
header, appending WTX Developer A’s app data and sending it to the MQ queue.
3.Developer B was not specifying the MQMD Encoding or the MQMD CCSID
fields when creating the message. I consider this is normal. In
general you want to allow these fields to default because by default
the MQ Client libraries will correctly identify the platform they are
executing on and will fill in the Encoding and CCSID appropriate for
the client platform. In this case the client platform is Datapower and
the default encoding value is 546 (little endian based on most likely
x86 chips) and the default encoding is 819 (a Unix variant), so that’s what the default values ended up being.
4.The encoding value is meant to describe the non character data like
integer bytes, and the CCSID describes the character data.
5.We now had an MQ message where the MQMD header had MQMD.Encoding 546
and MQMD.CCSID 819.
6.The MQIIH header that follows the MQMD was built to match whatever
the MQMD said. So the MQIIH’s character data was in the 819 code page.
The MQIIH also happens to have some integer byte fields, so the
encoding value in the header is relevant. The bytes were created to
match the MQMD.Encoding value of 546, so they were in little endian format. So far so code.
7.While the MQIIH has its own encoding and CCSID fields, they are not
used for anything. This is documented in the MQ manuals. This is
contrary to other types of headers like an RFH2 header, where the MQMD
describes the format of the RFH2, and then the RFH2 fields describe
what follows it, usually the app data. So the MQIIH and the app data
that follows it both need to be in the same encoding and CCSID.
8.In the application data created by the WTX Map, there was character data.
The WTX Map asked for “Native” data character set, so the character
data was produced in a character set appropriate for the executing
platform (DataPower) and this matched what the MQMD.CCSID field was defaulting to: 819. Still good.
9.In the application data created by the WTX Map, there were also some
binary integer fields in the data. This is not common at all but these
types of fields are required when sending to IMS via the OTMA bridge.
The WTX Map asked for the Byte Order of these fields to be “Big
Endian” (based on the UCFE model in step 1) and so the data was
produced in a big endian format. We now a message payload where some
of the fields are in big endian, but the MQMD.Encoding value is little
endian (546). This is what causes the abends on the mainframe. Some of
the data in the payload is not described correctly by the header in
this scenario and the next app that tries to MQ convert this data (the
RCVR MCA on the mainframe QM in this case) ends up with some of the fields completely garbled.
10.When Developer B hard coded the MQMD.Encoding field to 273 (big
endian) two things occurred. The MQIIH header was created with its
integer bytes in big endian, so it was still correctly described, but
now the message payload that was already in big endian was properly
described by the header as well. The conversion routine was now doing
its worked based on good information and the converted message was good. IMS was able to process it and send back a reply.
The lesson here is the vast majority of the time the data produced by
the MQ Client is in the CCSID and Encoding for the platform that the
MQ Client is executing on, and letting the MQMD.CCSID and
MQMD.Encoding default is just fine. But if your app has the capability
to produce data in a different encoding or CCSID you need to identify
that you are doing that and then set the MQMD fields so that the data is described correctly.
It looks like the WTX Map could set those integer bytes to the native
byte order which would give you little endian. If that was done the
default MQMD.Encoding of 546 would correctly describe the data and that would work.
A.The map sets the bytes to little endian and the MQMD.Encoding is
allowed to default to 546, or can be explicitly set to 546 for
clarity’s sake
B.The map set the bytes to big endian and the MQMD.Encoding must be
set to 273. This is how things are now.
I am not aware of there being a strong reason to go with A or B. If
writing this thing from the ground up I would probably vote for A. If
you are writing the request on a little endian platform, and the
receiver can do a standard convert, then produce the data in the
encoding that matches where the producer is executing. But B will work fine too.
*Peter Potkay*
Behalf Of *Potkay, Peter M (CTO Architecture + Engineering)
*Sent:* Thursday, June 20, 2013 8:35 AM
*Subject:* Re: DataPower, Encoding and the OTMA Bridge
…” I think you can agree that the CONVERT=YES is the way to go”
Nope J
I’ve had it beaten into my head every since my first week of MQ
training that receiver makes good, that channel conversion is an
option left over from the early days of MQ when certain MQ versions on
certain platforms could not convert some relatively common CCSIDs. If
one had to rely on a channel to convert, it was a band aid solution
trying to make up for a deficiency at one end or the other of the transaction.
In my case the sender and receiver are all current IBM software and hardware.
IBM DataPower with IBM WTX maps using IBM MQ Client to talk to an IBM
MQ queue manager which talks to an IBM IMS transaction via another IBM
MQ queue manager that runs on an IBM mainframe. All software current
and all infrastructure inside the USA, so no weird language
considerations. If I needed to rely on a SNDR channel to convert in this case I think it would be to work around a bug.
I don’t think there is an issue with conversion at this point. I think
it’s a matter of understanding how and why the CCSID and Encoding is
being set for the MQMD, the MQIIH and the actual data, and then tagging it appropriately.
Once the data is described accurately, conversion will work as
expected whether its by the receiver (the RCVR MCA on the mainframe in
this case because it’s a message destined for an OTMA queue – that was
a revelation during this research!) or a SNDR channel.
But I will give you this – your suggestion to use the SNDR channel for
conversion was crucial in my tests to come to the conclusions I have so far.
Seeing what the data looked like before and after conversion for
Message A and B, without having a receiving app involved, was key.
*Peter Potkay*
Behalf Of *Schanz, Arthur
*Sent:* Wednesday, June 19, 2013 4:06 PM
*Subject:* Re: DataPower, Encoding and the OTMA Bridge
Peter –
·I think you can agree that the CONVERT=YES is the way to go. It
produces the required EBCDIC data that the MF & IMS require.
·As far as the encoding, you can see that the value as shown in all of
the IIH samples is x’00000000’ (which is the numeric value of
MQENC_NATIVE), which is in the full word following the length field
<!-- CreateMQIIH headerto prefix the message data-->
<xsl:variable name="MQIIHNodeset">
<MQIIH>
<Encoding>0</Encoding>
<CodedCharSetId>1208</CodedCharSetId>
<Format>MQSTR</Format>
·For the payload, “Message B” works because the *LLZZ* is showing
valid values (the full word before the beginning of the IMS trancode).
This proves that this is using the necessary Encoding value.
Put these together and it is obvious that supplying the correct
Encoding value should do the trick.
Art
----------------------------------------------------------------------
----------
National IT Services*Arthur Schanz*
Distributed Computing Spec
Messaging and File Transfer
701 East Byrd Street
Richmond, VA 23219
Phone: 804.697.3889
Cell: 804.640.3132
National IT Green Logo
CertWS_color
Behalf Of *Potkay, Peter M (CTO Architecture + Engineering)
*Sent:* Wednesday, June 19, 2013 12:18 PM
*Subject:* Re: DataPower, Encoding and the OTMA Bridge
*Message A*
Put by DataPower (MQ Client 7.0.1.1) to a Red Hat Linux QM (MQ
7.0.1.6) without specifying the MQMD Encoding on the MQPUT. The
message ends up with an Encoding of 546, the CCSID is 819. The app
http://publib.boulder.ibm.com/infocenter/wsdatap/v4r0m2/topic/com.ibm.
dp.xb.doc/integratingwithmq39.htm?path=5_3_1_3_2_1#mqint_imsinfoheader
_concept
The first line of the MQIIH is as follows. This is a copy and paste
from BMTM looking at the data with the Text and Hex columns turned on.
*Code:*
I I H T 49494820 01000000 54000000 00000000
*Code:*
A 3 D H 8 7 0 1 C O N 00170000 41334448 38373031 20434F4E
*Message B*
Put by DataPower (MQ Client 7.0.1.1) to a Red Hat Linux QM (MQ
7.0.1.6) specifying the MQMD Encoding as 273 on the MQPUT. The message
ends up with an Encoding of 273, the CCSID is 819.
The first line of the MQIIH is as follows, first the text and then in hex.
Notice the difference in endianess of the 2nd and 3rd block of data
when compared to Message A.
*Code:*
I I H T 49494820 00000001 00000054 00000000
The first line of the message payload, after the MQIIH, is as follows.
Notice the bytes are exactly the same when compared to Message A, but
the MQMD has a different Encoding describing these exact bytes. Uh oh,
you can see there is going to be a conversion issue.
*Code:*
A 3 D H 8 7 0 1 C O N 00170000 41334448 38373031 20434F4E
http://publib.boulder.ibm.com/infocenter/wmqv7/v7r0/topic/com.ibm.mq.c
sqzak.doc/fr12840_.htm
This says the Encoding value of the MQIIH is taken from the preceding
header, in this case the MQMD.
It seems the MQMD Encoding value on the MQPUT is affecting the way the
bytes are produced in the MQIIH Header, but it is not having an effect
on the bytes in the message payload?
Repeating the test but instead having both Message A and B go from the
Linux QM to the mainframe QM over a SNDR channel with convert set to
NO has the messages looking exactly the same as when they were at rest
in a local queue on that Linux QM.
I change the SNDR channel on that Linux QM to have convert set to Yes
and repeat the test.
*Message A* now has an Encoding of 785 and a CCSID of 500. The first
*Code:*
I I H è C9C9C840 00000001 00000054 00000000
*Code:*
A 3 D H 8 7 0 1 C O N 0D400000 C1F3C4C8 F8F7F0F1 40C3D6D5
*Message B* now also has an Encoding of 785 and a CCSID of 500. The
*Code:*
I I H è C9C9C840 00000001 00000054 00000000
*Code:*
‡ A 3 D H 8 7 0 1 C O N 00170000 C1F3C4C8 F8F7F0F1 40C3D6D5
After conversion the bytes now differ between Message A and Message B
in the data payload, but the MQIIH looks the same. So the MQIIH header
is getting described properly by the MQMD Encoding and thus being
converted properly. The message payload beyond the MQIIH is
potentially not being described properly in one of these cases.
OK, lets see how the IMS application deals with this. Datapower
connects in client mode to that Linux Queue Manager putting to a
remote q that aims at the mainframe over a SNDR channel with Convert set to NO.
Message A causes an Abend and 3 MQ replies come back from the OTMA bridge.
Bunch of mainframe gibberish, put the phrase “NO SEGMENT RECEIVED
AFTER TCODE SEGMENT” is in one of the 3 reply messages.
Message B causes a successful reply message to come back.
I set the SND channel to convert, restart the channel and repeat the
test. No change – Message A causes an Abend with 3 replies and Message B works correctly.
So, I’m open to opinions, but here’s where I’m at. If the bytes in the
payload of the app data look exactly the same coming out of Datapower
regardless of the Encoding, but after translation they look different
when compared to each other, that says to me one of the Encodings is
not describing the data correctly on output. Based on testing it would
http://publib.boulder.ibm.com/infocenter/wmqv7/v7r0/topic/com.ibm.mq.c
sqzaq.doc/fc_MQENC_.htm
It describes the systems for the various encodings. It shows 546 for
Linux and Windows, and 273 for Unix. My testing seems to indicate that
273 is correct for DataPower and 546 is incorrect. If true, is this a
problem in the DataPower layer, or perhaps in the MQ Client version
(7.1.0.1) loaded into this firmware version of DataPower? Maybe its
picking up a Linux Encoding value, maybe from the Linux QM its
connected to? Picking up the queue manager’s encoding is contrary to
how MQ Clients should set an MQMD Encoding when asking for the
default. They should get the encoding of THEIR platform (not the QM’s)
because THEY are producing the bytes in the message. Or maybe the WTX
map is getting in the way and producing the data in a 273 encoding when it should be a 546 encoding?
As a test we connected Datapower directly to a z/OS Queue manager that
has an Encoding of 785. DataPower produced a message without setting
the MQMD Encoding and the message landed in the queue with an MQMD
encoding of 546 again. It did not (incorrectly) pick up the 785 from the z/OS Queue Manager.
Putting the message while connected to the z/OS QM and specifying the
273 encoding also produced the same message and MQIIH as Message B above.
I guess this means that the MQ Client 7.0.1.1 on a DataPower appliance
is populating the MQMD.Encoding with 546 if allowed to default. The
encoding of the MQIIH header is produced to match this 546 encoding,
but the message payload produced needs an encoding of 273. Shouldn’t
the encoding in the MQMD default to match the encoding of the data
being produced by the platform? If yes, is this a problem worthy of a
PMR? If yes, do you think it needs to go to IBM MQ or IBM DataPower or
IBM WTX? I want to make sure that by setting the MQMD.Encoding to 273
on the put in DataPower is a legit fix and not some hack work around that will byte us in the future.
**
**
*Peter Potkay*
Behalf Of *Potkay, Peter M (CTO Architecture + Engineering)
*Sent:* Monday, June 17, 2013 11:13 AM
*Subject:* Re: DataPower, Encoding and the OTMA Bridge
That is a good idea Art. I’ll set up a separate path in the test
environment and have the SNDR channel on the Linux QM do the convert.
We’ll have the DP appliance comment out the part where they set the
MQMD’s Encoding to 273 and let it default to 546. And then we’ll see
what the message looks like when it arrives on a plain local queue on
the mainframe side. If it arrives with a 273 encoding and the endian
bits flipped around correctly this will point to their being a problem
on the mainframe side with conversion of a message with 546 encoding.
*Peter Potkay*
Behalf Of *Schanz, Arthur
*Sent:* Monday, June 17, 2013 10:32 AM
*Subject:* Re: DataPower, Encoding and the OTMA Bridge
Peter –
We found early on in our OTMA ‘journey’ that separating the OTMA-bound
traffic from other non-OTMA traffic was the best for all involved.
This allowed us to use the channel/MCA to do the conversion, as the
msgs will go immediately from the receiving MCA, to the queue defined
with the XCF Storage Class, and across the OTMA bridge into IMS –
hence the comment about no ‘application’ having the ability to do an MQGET (and therefore no chance to do an MQGET w/ Convert).
Our OTMA msgs come (primarily) from CCSID 819 defined systems. We
are not requiring those applications to do anything ‘special’ to
generate the msgs – just ensure that they use the native CCSID/Encoding for the originating system.
If at all possible, I suggest you segregate your OTMA traffic from
your non-OTMA traffic…or at least run a quick test w/ a new channel
has CONVERT=YES set.
Cheers,
Art
----------------------------------------------------------------------
----------
National IT Services*Arthur Schanz*
Distributed Computing Spec
Messaging and File Transfer
701 East Byrd Street
Richmond, VA 23219
Phone: 804.697.3889
Cell: 804.640.3132
National IT Green Logo
CertWS_color
Behalf Of *Potkay, Peter M (CTO Architecture + Engineering)
*Sent:* Monday, June 17, 2013 10:09 AM
*Subject:* Re: DataPower, Encoding and the OTMA Bridge
Hi Art,
The request message in this case does have an MQIIH.Format of
MQFMT_IMS_VAR_STRING. But for some reason an incoming MQMD.Encoding of
546 causes issues, while a value of 273 works OK. In either case the
CCSID of 819 seems to be dealt with correctly for the character portion of the message.
With a 546 encoding the program pulling the messages of the IMS queue
abends complaining about missing or incomplete segments. I think
because the LL portion of the LLZZ, that describes the length, is not
interpreted correctly with an MQMD.Encoding of 546.
Setting the QM to QM channel to convert is not something I want to entertain.
There are tons of applications sharing this path between mid tier and
z/OS and many of them do not use the OTMA Bridge, and some successfully do.
·You mention “We use CCSID 037 (x'25')/ Encoding 785 (x'311') for our
traffic from distributed to z/OS”. Can you expand on that – are you
putting the onus on the mid tier app to produce an EBCIDIC request message?
I’m searching trying to get a clear answer on how messages from a
distributed queue manager arriving on a z/OS queue manager’s OTMA queue get converted.
http://publib.boulder.ibm.com/infocenter/wmqv7/v7r0/topic/com.ibm.mq.c
sqzal.doc/fg15970_.htm
/“The data conversion is performed by either the distributed queuing
facility ( which may call any necessary exits ) or by the intra group
queuing agent ( which does not support the use of exits ) when it puts
a message to a destination queue that has XCF information defined for
its storage class.”/
//
/“Because the WebSphere MQ-IMS bridge does not convert messages when
it gets a message….”/
I guess this is saying the RCVR MCA on a z/OS queue manager is slick
enough to know that it must do some conversion if an incoming message
bound for an MQ queue defined as an OTMA Bridge queue. And apparently
it is not doing it correctly or at all for an encoding of 546 but
since the 273 encoding is mainframe friendly (assumption on my part)
it works. Further assumption is that it has no issues converting the
character data from CCSID 819 of the request message to the CCSID 500 of our mainframe QM.
http://publib.boulder.ibm.com/infocenter/wmqv7/v7r0/topic/com.ibm.mq.a
mqwak.doc/ir11680_.htm
“•/Applications that connect to other queue managers can provide an
MQIIH structure that is in any of the supported character sets and
encodings; conversion of the MQIIH is performed by the receiving
message channel agent connected to the queue manager that owns the IMS
bridge queue. /
/Note: There is one exception to this. If the queue manager that owns
the IMS bridge queue is using CICS® for distributed queuing, the MQIIH
must be in the character set and encoding of the queue manager that owns the IMS bridge queue/.”
Further evidence that the RCVR MCA should be doing this. Guess the
question on the table is if 546 is a “supported encoding”.
I don’t know what “/If the queue manager that owns the IMS bridge
queue is using CICS® for distributed queuing/” means. We do have CICS
attached to this mainframe queue manager and unrelated incoming MQ
messages do trigger some CICS transactions. But I’m going to guess
this is not an issue because the character portion of the messages
that we send to the bridge queue is in an
819 CCSID and it is being dealt with.
Comments on the assumption that the RCVR MCA on our 7.0.1 z/OS Queue
Manager is not dealing with a 546 encoding correctly?
*Peter Potkay*
Behalf Of *Schanz, Arthur
*Sent:* Monday, June 17, 2013 8:57 AM
*Subject:* Re: DataPower, Encoding and the OTMA Bridge
Resending to correct horrible formatting of previous response and to
add comment about channel conversion…
Peter -
·Use the MQIMSVS format for msgs destined for the OTMA bridge.
(http://www-01.ibm.com/support/docview.wss?uid=swg21607923)
·Remember that the MQIIH structure is fixed length (84 bytes, or
x'54') and it is part of the payload or user data portion of the message.
·Messages need to be in the expected format for IMS to successfully
process
them: LLZZ TRANCODE DATA
·We use CCSID 037 (x'25')/ Encoding 785 (x'311') for our traffic from
distributed to z/OS
·Enable character set conversion in the channel via the CONVERT=YES
parameter in your channel definition. Since there is no application
prior to the OTMA bridge doing an 'MQGET', let the channel do the work.
These are some of the 'gotchas' that I have come across and should get
you going in the right direction.
Good Luck!
Art
Image removed by sender.
Image removed by sender.
*Arthur Schanz*
Distributed Computing Spec
Messaging and File Transfer
701 East Byrd Street
Richmond, VA 23219
Phone: 804.697.3889
Cell: 804.640.3132
----------------------------------------------------------------------
----------
List Archive <http://listserv.meduniwien.ac.at/archives/mqser-l.html>
- Manage Your List Settings
<http://listserv.meduniwien.ac.at/cgi-bin/wa?SUBED1=mqser-l&A=1> -
Unsubscribe
gnoff%20mqseries>
Instructions for managing your mailing list subscription are provided
in the Listserv General Users Guide available at http://www.lsoft.com
<http://www.lsoft.com/resources/manuals.asp>
************************************************************
This communication, including attachments, is for the exclusive use of
addressee and may contain proprietary, confidential and/or privileged
information. If you are not the intended recipient, any use, copying,
disclosure, dissemination or distribution is strictly prohibited. If
you are not the intended recipient, please notify the sender
immediately by return e-mail, delete this communication and destroy all copies.
************************************************************
----------------------------------------------------------------------
----------
List Archive <http://listserv.meduniwien.ac.at/archives/mqser-l.html>
- Manage Your List Settings
<http://listserv.meduniwien.ac.at/cgi-bin/wa?SUBED1=mqser-l&A=1> -
Unsubscribe
gnoff%20mqseries>
Instructions for managing your mailing list subscription are provided
in the Listserv General Users Guide available at http://www.lsoft.com
<http://www.lsoft.com/resources/manuals.asp>
----------------------------------------------------------------------
----------
List Archive <http://listserv.meduniwien.ac.at/archives/mqser-l.html>
- Manage Your List Settings
<http://listserv.meduniwien.ac.at/cgi-bin/wa?SUBED1=mqser-l&A=1> -
Unsubscribe
gnoff%20mqseries>
Instructions for managing your mailing list subscription are provided
in the Listserv General Users Guide available at http://www.lsoft.com
<http://www.lsoft.com/resources/manuals.asp>
************************************************************
This communication, including attachments, is for the exclusive use of
addressee and may contain proprietary, confidential and/or privileged
information. If you are not the intended recipient, any use, copying,
disclosure, dissemination or distribution is strictly prohibited. If
you are not the intended recipient, please notify the sender
immediately by return e-mail, delete this communication and destroy all copies.
************************************************************
----------------------------------------------------------------------
----------
List Archive <http://listserv.meduniwien.ac.at/archives/mqser-l.html>
- Manage Your List Settings
<http://listserv.meduniwien.ac.at/cgi-bin/wa?SUBED1=mqser-l&A=1> -
Unsubscribe
gnoff%20mqseries>
Instructions for managing your mailing list subscription are provided
in the Listserv General Users Guide available at http://www.lsoft.com
<http://www.lsoft.com/resources/manuals.asp>
************************************************************
This communication, including attachments, is for the exclusive use of
addressee and may contain proprietary, confidential and/or privileged
information. If you are not the intended recipient, any use, copying,
disclosure, dissemination or distribution is strictly prohibited. If
you are not the intended recipient, please notify the sender
immediately by return e-mail, delete this communication and destroy all copies.
************************************************************
************************************************************
This communication, including attachments, is for the exclusive use of
addressee and may contain proprietary, confidential and/or privileged
information. If you are not the intended recipient, any use, copying,
disclosure, dissemination or distribution is strictly prohibited. If
you are not the intended recipient, please notify the sender
immediately by return e-mail, delete this communication and destroy all copies.
************************************************************
************************************************************
This communication, including attachments, is for the exclusive use of
addressee and may contain proprietary, confidential and/or privileged
information. If you are not the intended recipient, any use, copying,
disclosure, dissemination or distribution is strictly prohibited. If
you are not the intended recipient, please notify the sender
immediately by return e-mail, delete this communication and destroy all copies.
************************************************************
To unsubscribe, write to ***@LISTSERV.MEDUNIWIEN.AC.AT and, in the message body (not the subject), write: SIGNOFF MQSERIES Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com
This communication, including attachments, is for the exclusive use of addressee and may contain proprietary, confidential and/or privileged information. If you are not the intended recipient, any use, copying, disclosure, dissemination or distribution is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return e-mail, delete this communication and destroy all copies.