Hi Damian,
My truss trace debugging suggestion would be more appropriate if this was a repeatable scenario. If this was a one off occurrence, then it would not be applicable.
I agree that going to latest fixpack level is a good idea.
Regarding the question,
"What role does the amqrmppa process play wrt a svrconn channel activity? Could this kind of traffic volume cause the process to use this much swap?"
I don't have enough understanding of the internal workings of MQ to answer that question authoritatively. But 15 gig of virtual memory for an MQ process is excessively high and way out of line based on my experience of supporting distributed MQ.
Just curious, did you happen to capture how much physical memory this amqrmppa process was using? The ps command for doing this (on Solaris/Linux) for a process with a pid of 1234 would be something like the following (vsz is the virtual memory, rss is the physical memory):
ps -p 1234 -o vsz, rss
Thanks,
Tim
-----Original Message-----
From: MQSeries List [mailto:MQSERIES-0lvw86wZMd9k/bWDasg6f+***@public.gmane.org] On Behalf Of Costa, D. (Damian)
Sent: Wednesday, October 09, 2013 2:32 PM
To: MQSERIES-0lvw86wZMd9k/bWDasg6f+***@public.gmane.org
Subject: Re: amqrmppa process using 15 gigs on virtual memory on a dev AIX box
Hi Tim,
I shut down the qmgr (killed it in fact). So clearly the qmgr was in some state of distress and could not shut itself down.
No user exits are involved.
Restarted it and monitoring the memory footprint on the amqrmppa process using topas. It started at 3.14 Megs, currently up to 3.48 Megs since restart ( 2 hours ago) .
Only app connected is the foglight agent which has moved 300k messages across the channel in since restart 2 hours ago.
This seems excessive to me just for monitoring purposes.
What role does the amqrmppa process play wrt a svrconn channel activity? Could this kind of traffic volume cause the process to use this much swap?
I'm inclined to just upgrade to 7.1.0.3 and go from there. No sense in wondering about a V7.1 level where there is potentially a high bug rate.
Post by Michael Dag-----Original Message-----
Behalf Of Tim Zielke
Sent: 09 October 2013 08:00 PM
Subject: Re: amqrmppa process using 15 gigs on virtual memory on a dev AIX box
Hi Damian,
15 gig of virtual storage is really high for an MQ process. At least based on what
I see for our Unix queue managers. Do you have any exit code running inside
the amqrmppa process?
You probably could narrow down the culprit with a system call trace. On AIX, I
believe it is a truss command. Assuming this is an environment where running a
truss on a queue manager would be appropriate, you could do the following to
start the queue manager (maybe with the assistance of one of the AIX
administrators if you are not familiar with how to run or read a truss).
truss -f strmqm qmgrname > truss.out 2>&1
Once the amqrmppa process gets to 15 gig or so, you can stop the trace. Inside
the truss trace there should be evidence of which thread (or threads) is
consuming this large amount of memory by analyzing the system calls that
involve memory allocation. Hopefully, something obvious stands out on which
thread is doing this. If you go to the beginning of the trace and see where this
thread is created, you should more than likely see the program name that the
thread is running under with the exec system call shortly after the thread was
created. Most threads are created with a fork/exec in Unix.
Thanks,
Tim Zielke
CICS/MQ Systems Programmer
Aon
-----Original Message-----
Behalf Of Costa, D. (Damian)
Sent: Wednesday, October 09, 2013 12:11 PM
Subject: Re: amqrmppa process using 15 gigs on virtual memory on a dev AIX box
Yup I think we'll be rolling V7.1.0.3 into development ASAP.
Just as a precaution.
Post by Michael Dag-----Original Message-----
Behalf Of Michael Dag
Sent: 09 October 2013 06:38 PM
Subject: Re: amqrmppa process using 15 gigs on virtual memory on a dev AIX box
Are you sure about 7.1.0.0, the latest is 7.1.0.3 there are a number
of amqrmppa and memory related fixes in there
Michael
-----Original Message-----
Behalf Of Costa, D. (Damian)
Sent: woensdag 9 oktober 2013 17:35
Subject: amqrmppa process using 15 gigs on virtual memory on a dev AIX box
Hi all,
We are evaluating MQ V7.1 on AIX v 7.1.0.0 The aix admins have
informed me that the amqrmppa process is using 15 gig's of virtual memory.
This would appear to be a crazy situation.
I think it's relate dot the monitoring agent Foglight in this case as
it has sent more data traffic to this qmgr than all the test harnesses
put together by a 99.99% differential. I dropped the client channel
that foglight is using to connect and monitor to see if that
alleviates the situation. I imagine this will not be the case.
Any suggestions? Short of bouncing the qmgr?
Perhaps the monitoring interval is too short?
Ta.
********************
Nedbank Limited Reg No 1951/000009/06. The following link displays the
names of the Nedbank Board of Directors and Company Secretary.
[ http://www.nedbank.co.za/terms/DirectorsNedbank.htm ] This email is
confidential and is intended for the addressee only.
The following link will take you to Nedbank's legal notice.
[ http://www.nedbank.co.za/terms/EmailDisclaimer.htm ]
********************
the message body (not the subject), write: SIGNOFF MQSERIES
Instructions for managing your mailing list subscription are provided
in the Listserv General Users Guide available at http://www.lsoft.com
Archive: http://listserv.meduniwien.ac.at/archives/mqser-l.html
the message body (not the subject), write: SIGNOFF MQSERIES
Instructions for managing your mailing list subscription are provided
in the Listserv General Users Guide available at http://www.lsoft.com
Archive: http://listserv.meduniwien.ac.at/archives/mqser-l.html
********************
Nedbank Limited Reg No 1951/000009/06. The following link displays the names
of the Nedbank Board of Directors and Company Secretary.
[ http://www.nedbank.co.za/terms/DirectorsNedbank.htm ] This email is
confidential and is intended for the addressee only.
The following link will take you to Nedbank's legal notice.
[ http://www.nedbank.co.za/terms/EmailDisclaimer.htm ]
********************
message body (not the subject), write: SIGNOFF MQSERIES Instructions for
managing your mailing list subscription are provided in the Listserv General
Users Guide available at http://www.lsoft.com
Archive: http://listserv.meduniwien.ac.at/archives/mqser-l.html
message body (not the subject), write: SIGNOFF MQSERIES Instructions for
managing your mailing list subscription are provided in the Listserv General
Users Guide available at http://www.lsoft.com
Archive: http://listserv.meduniwien.ac.at/archives/mqser-l.html
********************
Nedbank Limited Reg No 1951/000009/06. The following link displays
the names of the Nedbank Board of Directors and Company Secretary.
[ http://www.nedbank.co.za/terms/DirectorsNedbank.htm ]
This email is confidential and is intended for the addressee only.
The following link will take you to Nedbank's legal notice.
[ http://www.nedbank.co.za/terms/EmailDisclaimer.htm ]
********************
To unsubscribe, write to LISTSERV-0lvw86wZMd9k/bWDasg6f+***@public.gmane.org and,
in the message body (not the subject), write: SIGNOFF MQSERIES
Instructions for managing your mailing list subscription are provided in
the Listserv General Users Guide available at http://www.lsoft.com
Archive: http://listserv.meduniwien.ac.at/archives/mqser-l.html
To unsubscribe, write to LISTSERV-0lvw86wZMd9k/bWDasg6f+***@public.gmane.org and,
in the message body (not the subject), write: SIGNOFF MQSERIES