Discussion:
upgrade from 7.0.1.4 to 7.5.0.1 - all SVRCONN connect attempts get 2035
Potkay, Peter M (CTO Architecture + Engineering)
2013-10-11 15:47:41 UTC
Permalink
Dave,
Did you ever get to the bottom of this?

Peter Potkay


From: MQSeries List [mailto:***@LISTSERV.MEDUNIWIEN.AC.AT] On Behalf Of David Awerbuch (BLOOMBERG/ 120 PARK)
Sent: Friday, September 13, 2013 10:58 AM
To: ***@LISTSERV.MEDUNIWIEN.AC.AT
Subject: Re: upgrade from 7.0.1.4 to 7.5.0.1 - all SVRCONN connect attempts get 2035

Hi Peter,

Nope, never got a 2035 on this; all was working pre-upgrade. For instance, we have a package using the 7.0.1.4 wmq client that connects to each queue manager and retrieves the object definitions for recovery purposes. That package runs everyday, and generates exception reports. The first execution after the upgrade threw exceptions (2035) on the upgraded queue managers on that host.

as a rule, we do not disable OAM; we rely on it on dev to ensure that developers from different teams only have access to their queue objects, and can not accidentally put or get messages using queue objets that they do not own. Yes, there are some common userids as well for integration testing in DEV; we cover those with OAM as well.

we're racking our brains on this as well. Everything was working fine. we simply upgraded the software, which in turn upgraded the queue manager (at strmqm time). we changed nothing else - that's is our standard process in dev environment; each step is done individually in order to trap issues and isolate problems to the individual step. from lessons learned in DEV, we update our automated processes and prove them on UAT before they even go near PROD; odds are we learn new things in UAT and update the processes accordingly.

We are very specifically not implementing new features of 7.5 that need to be turned on and configured, we will do these at a later time again after experimenting in DEV and proving in UAT; at this point we want to take advantage of current code-base and some of the new multiple cluster channel features.

I've opened the PMR with IBM. Have to provide them with more info. I will try to keep everyone informed of the resolution and other important facts that come out of this PMR.

Thanks for your assist. It is always appreciated.

Dave

----- Original Message -----
From: ***@LISTSERV.MEDUNIWIEN.AC.AT<mailto:***@LISTSERV.MEDUNIWIEN.AC.AT>
To: ***@LISTSERV.MEDUNIWIEN.AC.AT<mailto:***@LISTSERV.MEDUNIWIEN.AC.AT>
At: Sep 13 2013 10:47:26
Did you ever get any 2035s on this QM before this upgrade? Maybe you had OAM disabled because it’s a DEV server with lax rules, and somehow the upgrade reenabled the OAM? Really grasping at straws here




Peter Potkay

From: MQSeries List [mailto:***@LISTSERV.MEDUNIWIEN.AC.AT] On Behalf Of David Awerbuch (BLOOMBERG/ 120 PARK)
Sent: Thursday, September 12, 2013 2:26 PM
To: ***@LISTSERV.MEDUNIWIEN.AC.AT<mailto:***@LISTSERV.MEDUNIWIEN.AC.AT>
Subject: Re: upgrade from 7.0.1.4 to 7.5.0.1 - all SVRCONN connect attempts get 2035

To reply to both of these posts -

1. Peter: "I still don’t understand why Dave is having the problem he is." -- Thank you!! I thought I was crazy, but Peter just reaffirmed I am not -- at least when it comes to the issue at hand.

2. None of the clients anywhere were upgraded. They are 7.0.1.4 - the user application (case 1) is Linux, the admin tool (case 2) is Windows. The only steps taken were to upgrade the server software and then the individual queue managers to 7.5.0.1; no other changes were made. That's how you are supposed to test new software - change only one component at a time and leave everything else alone; if the upgrade does some config updates, these must be accounted for -- which is the case here, I am not able to account for this change in authorization.

3. User application (Case 1) is c/c++; admin tool (Case 2) is apparently Java - lots of jar files - but I am verifying with the vendor.

4. channel not found errors were extracted from the AMQERR01.LOG as generated by the upgraded WMQ server. The channels exist, their definitions were not changed by us - I am also not able to account for these errors.

5. T.Rob, I changed the channel names (all names, actually) in my original note for security purposes. The real channels, however, are correct and continue to exist.

6. If I were connecting to a different queue manager than expected, than I would not see these errors in the correct queue manager, I would see them in a different queue manager.


To re-iterate:
1. queue manager upgraded in-place
2. clients not upgraded
3. no config or other hosting or infrastructure changes


I remain as perplexed as you do.

Dave

----- Original Message -----
From: ***@LISTSERV.MEDUNIWIEN.AC.AT<mailto:***@LISTSERV.MEDUNIWIEN.AC.AT>
To: ***@LISTSERV.MEDUNIWIEN.AC.AT<mailto:***@LISTSERV.MEDUNIWIEN.AC.AT>
At: Sep 12 2013 09:43:27
Well, the last question seems obvious
 The only way to get channel not found is 1) Someone defined SYSTEM.ADMIN.SVRCONN on the new QMgr where ADMIN.SVRCONN was expected; or 2) it's connecting to a different QMgr than expected. To even get that message the client had to have found *some* QMgr so it's the right one with the wrong definitions or the wrong one. The differences in behavior are hard to explain if the *only* action was to upgrade the QMgr in place. So I'm guessing there is more here we don't know yet – such as the QMgr was built on a new machine and moved over, or that clients were updated, etc.


From: MQSeries List [mailto:***@LISTSERV.MEDUNIWIEN.AC.AT] On Behalf Of Potkay, Peter M (CTO Architecture + Engineering)
Sent: Thursday, September 12, 2013 8:02 AM
To: ***@LISTSERV.MEDUNIWIEN.AC.AT<mailto:***@LISTSERV.MEDUNIWIEN.AC.AT>
Subject: Re: upgrade from 7.0.1.4 to 7.5.0.1 - all SVRCONN connect attempts get 2035

I still don’t understand why Dave is having the problem he is.

From his description I assume both Case 1 and Case 2 are MQ Clients that were not themselves upgraded, therefore I assume any identifying data they were or were not sending to the QM has not changed. So why would upgrading only the queue manager server cause these 2035s all of the sudden?

Dave, are these clients Java based? Was the MQ Client upgraded as well, or just the MQ Server? What’s up with those channel not found errors – that makes even less sense if the only thing that changed was the QM’s MQ version.


Peter Potkay

From: MQSeries List [mailto:***@LISTSERV.MEDUNIWIEN.AC.AT] On Behalf Of T.Rob
Sent: Wednesday, September 11, 2013 4:47 PM
To: ***@LISTSERV.MEDUNIWIEN.AC.AT<mailto:***@LISTSERV.MEDUNIWIEN.AC.AT>
Subject: Re: upgrade from 7.0.1.4 to 7.5.0.1 - all SVRCONN connect attempts get 2035

If the old connection attached as mqm, then set the MCAUSER('mqm') on the new channel. The old behavior of not sending a user ID in many cases was due to an insecure-by-default posture. It always allowed anonymous admin access, and was easily spoofed to be blank or mqm. The only connections using an actual valid ID were the very people you don't need to worry about. Setting MCAUSER('mqm') does not increase the exposure in the least, but it *does* make it more apparent that the QMgr leaks anonymous admin access. By reducing that false sense of security, people tend to be more aware of how exposed the thing is and thus act accordingly. Or somewhat more accordingly.
Note -- 7.5.0.1 queue manager has CHLAUTH(DISABLED)
Dave, you're KILLING me here. I have availability coming up if you want to actually fix this debacle, enable CHLAUTH and secure a few things. ☺

-- T.Rob



From: MQSeries List [mailto:***@LISTSERV.MEDUNIWIEN.AC.AT] On Behalf Of David Awerbuch (BLOOMBERG/ 120 PARK)
Sent: Wednesday, September 11, 2013 16:22 PM
To: ***@LISTSERV.MEDUNIWIEN.AC.AT<mailto:***@LISTSERV.MEDUNIWIEN.AC.AT>
Subject: upgrade from 7.0.1.4 to 7.5.0.1 - all SVRCONN connect attempts get 2035

I am hoping some one out there has already seen this issue during their upgrade cycle.

We upgraded a host with existing queue managers from 7.0.1.4 to 7.5.0.1. Our client applications are not able to connect to the queue managers using the existing svrconn channels and authorization records - 2035 errors.

1. User Application

Checked queue manager log, I see:

AMQ8077: Entity 'dbaseuser ' has insufficient authority to access object 'NEWSQMGR'.
AMQ9557: Queue Manager User ID initialization failed.

'dbaseuser' is an existing unix userid. until the upgrade to 7.5, the access through this srvconn was not an issue.
Note - the mcauser() field on the svrconn has always been and is blank.
I believe in the previous release (7.0.1.4) that this user would connect as user 'mqm'; thus no issues with connectivity and object access.

Now with 7.5.0.1 it looks like the real userid from the client connection header is being exposed to the authorization module. is this a correct assessment? Is there a way to restore this processing, or am I looking at a moving-forward series of changes to my application svrconns?

Note -- 7.5.0.1 queue manager has CHLAUTH(DISABLED)


2. WMQ Administration tool - web-based service

Checked queue manager log, I see this series of error messages; these are all generated by the same pid.tid:

AMQ9780: Channel to remote machine '1.2.3.4' is ending due to an error.
EXPLANATION:
Channel 'ADMIN.SVRCONN' between this machine and the remote machine
'1.2.3.4' encountered an error and will now end. In some cases the
channel name can not be determined and so is shown as '????'.

AMQ9519: Channel 'ADMIN.SVRCONN' not found.
EXPLANATION:
The requested operation failed because the program could not find a definition
of channel 'ADMIN.SVRCONN'.

AMQ9999: Channel 'ADMIN.SVRCONN' to host 'windowshost(1.2.3.4)'
ended abnormally.
EXPLANATION:
The channel program running under process ID 8392 for channel
'ADMIN.SVRCONN' ended abnormally. The host name is 'sysnyweb03
(1.2.3.4)'; in some cases the host name cannot be determined and so is
shown as '????'.

AMQ5653: The user 'windowshost$' is not defined.
EXPLANATION:
The system call getpwnam("windowshost$") failed with errno 2.

AMQ9557: Queue Manager User ID initialization failed.
EXPLANATION:
The call to initialize the User ID failed with CompCode 2 and Reason 2035.


Previous to the upgrade, this worked just fine.
Note - also in this case - the mcauser() field on the svrconn has always been and is blank.
so, the same premises and belief exists as above - in the previous release (7.0.1.4) this user would connect as user 'mqm'; thus no issues with connectivity and object access.

Now with 7.5.0.1 it looks like the machine name from the client connection header is being exposed to the authorization module; or the header contains the machinename with the appended '$'. Is this a correct assessment? Again, is there a way to restore this processing, or am I looking at a moving-forward series of changes to my admin svrconns?


Thanks.
Dave





------------------------------------------------------------
-------------------------------------------------------------------
David Awerbuch -- Systems Architect -- Middleware Infrastructure
email: ***@bloomberg.net<mailto:***@bloomberg.net> || ==== Bloomberg LP ==== ||
phone: + 1 646 324 2868 || Making Financial ||
WMQ support mq-***@bloomberg.com<mailto:mq-***@bloomberg.com> || Markets Transparent ||
-------------------------------------------------------------------
For all Trading Systems issues, please contact the TSCI 24-hour helpdesk.
- Americas - New York +1+212-318-2000
- Europe - London +44-20-7330-7500
- Asia - Tokyo +81-3-3201-8900
- Singapore +65-6212-1000


<< {IS} May you and yours be inscribed in the Book Of Life for health and happiness >>

________________________________
List Archive<http://listserv.meduniwien.ac.at/archives/mqser-l.html> - Manage Your List Settings<http://listserv.meduniwien.ac.at/cgi-bin/wa?SUBED1=mqser-l&A=1> - Unsubscribe<mailto:***@LISTSERV.MEDUNIWIEN.AC.AT?subject=Unsubscribe&BODY=signoff%20mqseries>

Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com<http://www.lsoft.com/resources/manuals.asp>

************************************************************
This communication, including attachments, is for the exclusive use of addressee and may contain proprietary, confidential and/or privileged information. If you are not the intended recipient, any use, copying, disclosure, dissemination or distribution is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return e-mail, delete this communication and destroy all copies.
************************************************************

________________________________
List Archive<http://listserv.meduniwien.ac.at/archives/mqser-l.html> - Manage Your List Settings<http://listserv.meduniwien.ac.at/cgi-bin/wa?SUBED1=mqser-l&A=1> - Unsubscribe<mailto:***@LISTSERV.MEDUNIWIEN.AC.AT?subject=Unsubscribe&BODY=signoff%20mqseries>

Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com<http://www.lsoft.com/resources/manuals.asp>


<< {IS} May you and yours be inscribed in the Book Of Life for health and happiness >>

************************************************************
This communication, including attachments, is for the exclusive use of addressee and may contain proprietary, confidential and/or privileged information. If you are not the intended recipient, any use, copying, disclosure, dissemination or distribution is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return e-mail, delete this communication and destroy all copies.
************************************************************


<< {IS} May you and yours be inscribed in the Book Of Life for health and happiness >>
************************************************************
This communication, including attachments, is for the exclusive use of addressee and may contain proprietary, confidential and/or privileged information. If you are not the intended recipient, any use, copying, disclosure, dissemination or distribution is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return e-mail, delete this communication and destroy all copies.
************************************************************
David Awerbuch (BLOOMBERG/ 120 PARK)
2013-10-11 15:50:44 UTC
Permalink
Hi Peter,

We have not yet determined a resolution for this issue. I opened a PMR for this with IBM, they continue to look at it, along with a second issue we raised related to it but not apparently the immediate cause.

I will post the results as soon as they are available.

Thanks,
Dave




<< Off to South Africa to see NASA's Juno streak by at 9:12 PM! When's my flight?? >>
Loading...