Hello,
Peter - Halloween . . . Yes, and Linux has zombie processes, too . . . :-)
The queue manager is usually in an unstable state when this happens. Potentially, one of its core processes has been terminated. From the queue manager error log stand point, I have seen FDC and AMQ error log entries about an MQ process being terminated unexpectedly. What the oom-killer is doing is very similar to going to your queue manager and doing a "kill -9" or sending the KILL signal to one of your queue manager processes. Here is an example of what you could see in the MQ error logs:
10/16/2013 10:04:30 PM - Process(18586.1) User(mqm) Program(amqzxma0)
Host(xxxxxxxx) Installation(Installation1)
VRMF(7.1.0.2)
AMQ5009: WebSphere MQ agent process 18643 has terminated unexpectedly.
If you want to see if the oom-killer caused this, you can search for oom in the kernel log file (i.e. /var/log/messages on our Linux systems). There will also be messages on which specific processes the oom-killer terminated. So you can trace it back to the MQ process that was terminated, if the oom-killer was the culprit.
The oom-killer is part of the Linux kernel. So it should be in any of the supported enterprise Linux releases like SLES, RHEL, etc. Doing some internet searches, it looks like the oom-killer came around in the 2.4 kernel. Must shops are probably using the 2.6 kernel. If you do a "uname -a", you can see what version of the Linux kernel you are running.
You can see if the oom-killer is enabled by doing a "sysctl -a | grep overcommit_memory". If that comes back as 0 or 1, the oom-killer is enabled. A value of 2 disables the oom-killer, but that also turns off the kernel from being able to overcommit on memory. So you can start to run into other system issues with running with that configuration, since the operating system now supports much less overall memory.
There are also ways to turn off the oom-killer from terminating a specific process by adjusting the /proc/pid#/oom_adj value to -17. But to me, this is kludgey and hard to operationalize.
We tried working with our Linux administrators on coming up with a good way to turn it off, and we ended up with just living with it. Fortunately, our Linux servers have enough memory now and we have also tuned the previous application memory consumption issues where this has not caused us grief in the recent past.
Thanks,
Tim
From: MQSeries List [mailto:***@LISTSERV.MEDUNIWIEN.AC.AT] On Behalf Of ***@COMCAST.NET
Sent: Wednesday, October 16, 2013 7:55 PM
To: ***@LISTSERV.MEDUNIWIEN.AC.AT
Subject: Re: WMQ on RHEL doco
...and is this process turned on or have the same behavior in the supported enterprise linux, such as RHEL?
I have never run into it yet...
________________________________
From: "Peter M Potkay (CTO Architecture + Engineering)" <***@THEHARTFORD.COM>
To: ***@LISTSERV.MEDUNIWIEN.AC.AT
Sent: Wednesday, October 16, 2013 8:42:26 PM
Subject: Re: WMQ on RHEL doco
Tim,
A timely post it being the Halloween season and all. What are the symptoms from an MQ perspective when the oom-killer grabs at a Queue Manager? Is there a particular ProbeID in an FDC to look for? Or a unique AMQ**** message in the error logs?
Peter Potkay
-----Original Message-----
From: MQSeries List [mailto:***@LISTSERV.MEDUNIWIEN.AC.AT] On Behalf Of Tim Zielke
Sent: Wednesday, October 16, 2013 12:00 PM
To: ***@LISTSERV.MEDUNIWIEN.AC.AT
Subject: Re: WMQ on RHEL doco
Hi Don,
I work with both Solaris 10 SPAC and Linux x86. I have found for the most part, that your Unix skills port over nicely to Linux.
If you use a lot of shell scripting, I've found some minor differences between Solaris and Linux. But again, it mostly ports over nicely.
For tips on Linux reference material, I would need to know more what you are specifically looking for. One nice thing about Linux is that it is open source, so there are a lot of books that do a deep dive into the kernel and how things work. Of course, you can, too :-). Anything by Robert Love is outstanding.
Also, be aware of the Linux oom-killer (out of memory killer). Linux (by default, I think) will overcommit on memory. So it will allow the kernel to get squeezed on physical memory. At some point the kernel says enough, and its default implementation is to release the oom-killer. This kernel process will run and look for processes with large amounts of memory to kill to free up the physical memory. Unfortunately, I have seen it pick on the MQ processes, eventhough they are very low in memory usage. There is something about the way they use memory (maybe shared memory?) that makes them a candidate for this kernel assassin and how it determines processes to kill. In my opinion, the oom-killer is an example of the risks of using an open source operating system that ultimately does not answer to customers. I have never seen any other commercial enterprise operating system do something of this nature. At least the ones I have worked with.
Tim Zielke
CICS/MQ Systems Programmer
Aon
-----Original Message-----
From: MQSeries List [mailto:***@LISTSERV.MEDUNIWIEN.AC.AT] On Behalf Of Thomas, Don
Sent: Wednesday, October 16, 2013 10:05 AM
To: ***@LISTSERV.MEDUNIWIEN.AC.AT
Subject: Re: WMQ on RHEL doco
Thanks Bruce, you're right, I was not at all clear on what I'm looking for. I have a client that is currently running WMQ V7 on Solaris and they are looking to migrate their installations to a virtual private cloud running RHEL. My involvement, at least at this time, will be administration of WMQ only. This will be my first dive into the Linux pool so any good tips on Linux reference material would be greatly appreciated as well.
-----Original Message-----
From: MQSeries List [mailto:***@LISTSERV.MEDUNIWIEN.AC.AT] On Behalf Of Bruce Lerner
Sent: Wednesday, October 16, 2013 10:09 AM
To: ***@LISTSERV.MEDUNIWIEN.AC.AT
Subject: Re: WMQ on RHEL doco
Documentation on what exactly? Linux? WMQ system admin? WMQ application development? I'd certainly add WebSphere MQ Script Command Reference (MQSC) to your reading list.
Search google for 'ibm+wmq+manuals'.
To unsubscribe, write to ***@LISTSERV.MEDUNIWIEN.AC.AT and, in the message body (not the subject), write: SIGNOFF MQSERIES Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com
Archive: http://listserv.meduniwien.ac.at/archives/mqser-l.html
To unsubscribe, write to ***@LISTSERV.MEDUNIWIEN.AC.AT and, in the message body (not the subject), write: SIGNOFF MQSERIES Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com
Archive: http://listserv.meduniwien.ac.at/archives/mqser-l.html
To unsubscribe, write to ***@LISTSERV.MEDUNIWIEN.AC.AT and, in the message body (not the subject), write: SIGNOFF MQSERIES Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com
Archive: http://listserv.meduniwien.ac.at/archives/mqser-l.html
************************************************************
This communication, including attachments, is for the exclusive use of addressee and may contain proprietary, confidential and/or privileged information. If you are not the intended recipient, any use, copying, disclosure, dissemination or distribution is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return e-mail, delete this communication and destroy all copies.
************************************************************
To unsubscribe, write to ***@LISTSERV.MEDUNIWIEN.AC.AT and,
in the message body (not the subject), write: SIGNOFF MQSERIES
Instructions for managing your mailing list subscription are provided in
the Listserv General Users Guide available at http://www.lsoft.com
Archive: http://listserv.meduniwien.ac.at/archives/mqser-l.html
________________________________
List Archive<http://listserv.meduniwien.ac.at/archives/mqser-l.html> - Manage Your List Settings<http://listserv.meduniwien.ac.at/cgi-bin/wa?SUBED1=mqser-l&A=1> - Unsubscribe<mailto:***@LISTSERV.MEDUNIWIEN.AC.AT?subject=Unsubscribe&BODY=signoff%20mqseries>
Instructions for managing your mailing list subscription are provided in the Listserv General Users Guide available at http://www.lsoft.com<http://www.lsoft.com/resources/manuals.asp>