WebSphere MQ

Monday, September 22, 2014

Problem in MQ v7.1 using permanent dynamic queues

IBM just release a "Flash Alert" on page set zero chain corruption using permanent dynamic queues.

After applying UI11858 or its superseded UI13085...some users may experience a chain corruption in PSID(0) causing a loop in a queue manager SRB.

Full details of the problem and its symptoms as well as abend details can be found here:

http://www-01.ibm.com/support/docview.wssuid=swg21684118&myns=swgws&mynp=OCSSFKSJ&mync=E

Happy messaging.

Sunday, February 17, 2013

Extending Shared Queues using Shared Message Datasets

In this blog entry I will talk about what Shared Message Datasets (SMDS) are, what problem(s) they solve and how to implement and manage SMDS in your Queue Sharing Group (QSG).

What are Shared Message Datasets?

Currently, shared message queues and their message data are stored in structures that are built in the Coupling Facility. These structures are defined with a fixed size, so all message queues assigned to a particular structure, are constrained by this size. Once a CF structure is full, the reason code 2192 is issued, indicating that the storage medium for that queue is full. In V7.1 new function mode, a new construct called a Shared Message Dataset has been introduced to prevent CF structures from filling up. The SMDS is a linear VSAM dataset much like a pageset for non-shared queues today, but is associated with a CF structure.

Implementing Shared Message Datasets

Before we can implement a SMDS architecture all queue managers in the queue sharing group need to have the OPMODE=(NEWFUNC ,710) set in the ZPARM. Once this has been set, and the queue managers have been restarted we can now allocate the Shared Message Datasets.

Each Queue Manager in the queue sharing group will have its own copy of the SMDS for a coupling facility structure. This is designated by the use of an * in the DSGROUP parameter of the CFSTRUCT definition.

See below:

Queue Manager Queue Manager

QMA1 QMB1

/ \

/ Queue Sharing Group \

/ (QM01) \

'MQVS.QM01.QMA1.SHAREQ.SMDS' 'MQVS.QM01.QMB1.SHAREQ.SMDS'

CFSTRUCT

(SHAREQ)

DSGROUP('MQVS.QM01.*.SHAREQ.SMDS')

You can see the DSGROUP attribute is MQVS.QM01.*.SHAREQ.SMDS. The queue manager will take the * and substitute its name for the SMDS it is to use.

Now, let’s create and format the shared message datasets we want to use. This is done using two utilities, IDCAMS and CSQJUFMT.

//CRESMDSA JOB (ABCD,1234),'J LANG',CLASS=C,MSGCLASS=H

//*

//***********************************************************

//* Allocate the SMDSs

//***********************************************************

//DEFINE EXEC PGM=IDCAMS,REGION=4M

//SYSPRINT DD SYSOUT=*

//SYSIN DD *

DELETE 'MQVS.QM01.QMA1.SHAREQ.SMDS' ERASE CLUSTER

SET MAXCC=0

DEFINE CLUSTER -

(NAME(MQVS.QM01.QMA1.SHAREQ.SMDS) -

MEGABYTES(200 300) -

LINEAR -

DATACLAS(DCXVSM) -

SHAREOPTIONS(2 3) ) -

DATA -

(NAME(MQVS.QM01.QMA1.SHAREQ.SMDS.DATA) )

//***********************************************************

//* Format the SMDS

//***********************************************************

//FORM EXEC PGM=CSQJUFMT,COND=(0,NE),REGION=0M

//STEPLIB DD DSN=MQM.SCSQANLE,DISP=SHR

// DD DSN=MQM.SCSQAUTH,DISP=SHR

//SYSUT1 DD DISP=OLD,DSN=MQVS.QM01.QMA1.SHAREQ.SMDS

//SYSPRINT DD SYSOUT=*

The job above is for the QMA1 queue manager…you will also need to change the QMA1 to QMB1 and run the job again. Also you will notice a DATACLAS above, this is telling IDCAMS that this VSAM file has extended addressability and can be expanded past the 4GB limit. It can actually grow to 16TB given that you have that amount of DASD available.

So now we have our shared message datasets for each of our queue managers in the queue sharing group, and we also have each queue manager running with the OPMODE=(NEWFUNC ,710). So we have two options, we can alter our current CTSTRUCT to Level 5 and set the appropriate variables, or we can create a new CFSTRUCT set to Level 5. If we are creating a new struct, work will need to be done in the coupling facility policy to allocate this new structure, and the new policy will need to be started to become active. For the second option you can get with your z/OS systems programmer and they will handle that portion.

For now, let’s look at altering a current CFSTRUCT.

It is easiest to do this in a batch job since several new attributes need to be set at the same time.

//SMDSJOBA JOB (ABCD,1234),'J LANG',

// CLASS=C,

// MSGCLASS=H

//*

//EXTRACT EXEC PGM=CSQUTIL,PARM='QMA1'

//STEPLIB DD DSN=MQM.SCSQANLE,DISP=SHR

// DD DSN=MQMQ.QMA1.SCSQAUTH,DISP=SHR

// DD DSN=MQM.SCSQAUTH,DISP=SHR

//SYSPRINT DD SYSOUT=*

//SYSIN DD *

COMMAND DDNAME(INPUT)

//*

//INPUT DD *

ALTER -

CFSTRUCT('SHAREQ') -

DESCR('Level 5 CFSTRUCT') -

CFLEVEL(5) -

RECOVER(YES) -

OFFLOAD(SMDS) see note 1 -

OFFLD1TH(70) see note 2 -

OFFLD1SZ(32K) “ -

OFFLD2TH(80) “ -

OFFLD2SZ(4K) “ -

OFFLD3TH(90) “ -

OFFLD3SZ(0K) “ -

DSGROUP('MQVS.QM01.*.SHAREQ.SMDS') -

DSBLOCK(256K) -

DSBUFS(100) -

DSEXPAND(YES) -

RECAUTO(NO) -

CFCONLOS(TERMINATE)

//*

Note 1: When changing the level of a CFSTRUCT to 5, the OFFLOAD parameter must be set to DB2 or SMDS. In our case, we are using shared message datasets due to the performance impact of using DB2. (this topic is for another paper)

Note 2: There are three sets of offload rules that the queue manager uses. These contain the percentage that the coupling facility structure must reach before the rule kicks in. In our listing above, we have the following:

OFFLD1TH(70) – When the coupling facility reaches 70% full

OFFLD1SZ(32K) – New messages greater than 32K go to the SMDS

OFFLD2TH(80) – When the coupling facility reaches 80% full

OFFLD2SZ(4K) – New messages greater than 4K go to the SMDS

OFFLD3TH(90) – When the coupling facility reaches 90% full

OFFLD3SZ(32K) – New messages greater than 0K go to the SMDS

Special Note: There is still some data stored in the coupling facility for a message that is offloaded to the SMDS, so the larger your messages are, the more benefit you will get from using SMDS. The 3^rd rule here, says that if the CF is 90% full, all new messages go to the SMDS. For messages that are under 140 bytes, you will not benefit from this rule, and the CF will still become 100% full because while it is moving let’s say 100 bytes messages to the SMDS, it is using more space than that to control the pointers to these messages.

So now we have our CF at level 5, and we have our SMDS allocated. You need to make sure the that the queue manager started task ID’s have UPDATE access to the SMDS VSAM datasets. (RACF Rules). The owning queue manager will have its SMDS open for UPDATE, other queue managers in the group will have all other SMDS’s open for READONLY.

There are several new commands that system programmers have available to them to aid in the management of the SMDS. Below are a few of the most common: (you will use your own queue manager name)

/-QMA1 DIS SMDSCONN(QMA1) CFSTRUCT(SHAREQ)

Display the status and availability to the SMDS by the queue manager.

Output:

CSQM201I –QMA1 CSQMDRTC DIS SMDSCONN DETAILS

SMDSCONN(QMA1)

CFSTRUCT(SHAREQ)

OPENMODE(UPDATE)

STATUS(OPEN)

AVAIL(NORMAL)

EXPANDST(NORMAL)

END SMDSCONN DETAILS

/-QMA1 START SMDSCONN(QMA1) CFSTRUCT(SHAREQ)

Start the connection to the SMDS from the queue manager. This will allocate and open

the SMDS if it is new, otherwise it automatically happens.

Output:

-QMA1 CSQMSSMD ' START SMDSCONN' NORMAL COMPLETION

The change you will see from the start will be shown by displaying the connection.

While running a job to load the queue, you will see expansion messages in the queue manager address space if it is allocating more space for the SMDS…this is much like normal page set expansion.

CSQE213I -QMA1 CSQEDSS2 SMDS(QMA1) CFSTRUCT(SHAREQ)

data set MQVS.QM01.QMA1.SHAREQ.SMDS is now 98% full

CSQE239I -QMA1 CSQEDSS2 SMDS(QMA1) CFSTRUCT(SHAREQ)

data set MQVS.QM01.QMA1.SHAREQ.SMDS has become full so new large

messages can no longer be stored in it

CSQE212I -QMA1 CSQEDSI1 Formatting is complete for

SMDS(QMA1) CFSTRUCT(SHAREQ) data set MQVS.QM01.QMA1.SHAREQ.SMDS

CSQE217I -QMA1 CSQEDSI1 Expansion of SMDS(QMA1)

CFSTRUCT(SHAREQ) data set MQVS.QM01.QMA1.SHAREQ.SMDS was successful

76860 pages added, total pages 128160

While this is happening, your job, or channels will be paused until the expansion completes.

Other SMDS commands: (for a complete list of parameters, see MQ Command Reference)

ALTER SDMS (qmgr|*) CFSTRUCT(cfname) DSBUFS(number)

DSEXPAND(yes|no|)

DISPLAY SMDS (qmgr|*) CFSTRUCT(cfname) WHERE(filter)

DISPLAY SMDSCONN (qmgr|*) CFSTRUCT(cfname) WHERE(filter)

RESET SMDS (qmgr or *) CFSTRUCT(cfname) ACCESS(enabled|disabled) STATUS(failed|recovered)

START SMDSCONN (qmgr or *) CFSTRUCT(cfname)

CMDSCOPE(‘ ‘|qmgr|*)

STOP SMDSCONN (qmgr or *) CFSTRUCT(cfname)

CMDSCOPE(‘ ‘|qmgr|*)

Optional parameter

Testing:

I had two CF structures established of the same size. One was at Level 4, one was at Level 5 with a SMDS assigned to the CFSTRUCT. I attempted to put 50,000 messages that were 40,000 bytes each. In the Level 4 CFSTRUCT, I was able to load 4710 messages in the queue before the RC 2192 was returned stating that the CF was full. In the Level 5 structure with the SMDS assigned, once the CF level reached 70%, the data started going to the SMDS and I was able to put all 50,000 messages on the shared queue.

Saturday, May 21, 2011

MQ Version 7 "gotcha" on z/OS

I have gone over the "WebSphere MQ V7.0 Features and Enhancements", the "Migration Guide"....and this was not mentioned once. I did find reference to this in SupportPac MP1G...but it was after the fact and I had already converted to V7.01.

In version 7.01 of MQ for z/OS....IBM changed the way messages are stored on the pageset. If a message is under 2K....it is stored 1 message per page. This was huge for one of my applications.....as they kept filling the pageset, and overflowing to the dead letter queue. It took a little while to figure this out. Regardless of the design implications....this application would get data all day, and then process it at night. Well it didn't take long to fill the pageset up to the 4GB limit. These messages are 153 bytes each.......so we use a 4K page to store 153 bytes. We went from approximately 20 messages per page....to one.

The solution was to allocate a new "extended" VSAM dataset to be able to go over the 4GB limit. Once this was allocated and formatted......I then dynamically added the pageset to the queue manager:

-qmgr DEFINE PSID(9) BUFFPOOL(3) DSN('MQM.qmgr.PSID09') EXPAND(USER)

Then I created a new STORAGE CLASS called EXPANDED to use this pageset...and finally, I had the application process all of the data...stopped all incoming channels so the queue stayed empty, and then reassigned the queue to the new storage class.

Remember when you add a pageset dynamically, you will still need to change the PROC to include the new DD statement for the pageset, as well as to update the CSQINP1 input dataset to include the pageset to buffer pool entry:

DEFINE PSID( 09 ) BUFFPOOL( 3 )

as well as the input for CSQINP2, where you assign the storage class:

DEFINE STGCLASS( 'EXTENDED') +
PSID( 09 )

There is also another work around that is talked about in the MP1G Performance SupportPac...and it is having this statement in your CSQINP2 DD statement:

REC QMGR (TUNE MAXSHORTMSGS 0)

This will have the queue manager store messages in the pre v7.01 way....more than one on a page.
It seems to me that this should have been pointed out in the migration guide, as well as mentioned in the features and enhancements book. Having to search for information on this after the fact, was not really what I had in mind.

Saturday, April 9, 2011

Don't Ignore your FDC Probes.

I cannot tell you how many client sites I have been at where they seem to have unexplainable behavior in their WebSphere MQ environments. The first question I ask them is "Have you looked at your FDC probes?" Most of the time they do not know what I am talking about. So I explain, that MQ will tell you when it does not like something in the environment, or is having some type of trouble. In some of my engagements, the MQ servers are running fine, at least there is no apparent problem, but when I look....I usually find some area where tuning is needed.

What I am referring to is the FDC files that MQ cuts when it has experienced a failure. On the distributed platforms..ie. Unix, Linux..etc. these are stored in /var/mqm/errors. Most of the time when an FDC event happens, the programs, services and channels will keep running, so it it not readily noticeable that a problem has occurred.

As MQ administrators, one of our functions is to monitor these files, trying to detect a pattern, or at least cataloging and quantifying what has happened. Armed with this information, we can make adjustments to our environments and get them running to full capacity.
So let's look at getting at the probes...and how many of which type occurred. To do this, we will use some simple Unix/Linux commands.

First off, change to the /var/mqm/errors directory:

$ cd /var/mqm/errors

Now, lets get the probe number, and a count of how many times each has happened:

$ grep "Probe Id" *.FDC|cut -c 38-47 | sort | uniq -c

This is just an example, but your output might look something like this:

      9 HL047028
      4 HL049110
      9 HL077070
     47 XC307070
     43 XC308010

You might have some lines that are offset and duplicates...but that is due to your FDC file name length and this is based on the process ID number. You can add them together for a total occurrence count.

Okay...these "Probe Id's" are coded based on the first two characters as to which MQ component had the failure. In our case, we have HL and XC. The HL is the "Logger" component and the XC is "Common Services". A partial list of these probe codes can be found on my website, http://www.tpmq-experts.com/FDC-Probes.html.

Okay...as we can see, we had 37 XC307070 (Long Lock Wait) probes and 43 XC308010 (Mutex Release) probes. This is not a catastrophic failure....but does indicate that you are using linear logging and are taking media images at a time when your queues are at a high depth. This does affect the speed of processing for your applications, so it would be good to have a check in your record image script to check the queue depth....if it is high, then it would be wise to wait to perform the record image.

Looking at the other probes, we had 9 HL047028 (Logger problem), 4 HL049110 (Logger problem) and 9 HL077070 (disk full). The last probe, is self explanatory...the disc was full. So if you are using Linear logging...you clean-up script needs to be adjusted, or more disk space allocated. The other two probes point at kernel parameters msgmnb, msgtql and msgseg..which can all be adjusted to handle the load that your current environment needs.

These are just a few examples of probes that can be corrected, which will have a positive impact on the running of your environment and the throughput of your applications.

Have a look at your probes...and really see what's going on.

Saturday, March 26, 2011

Sending, reading and receiving a WebSphere MQ message using SQL.

A little understood and exploited feature of WebSphere MQ and DB2, is the ability to send, read and receive a message using a simple SQL statement. The validity of using this feature we will leave for the systems architects and application developers. The purpose of this post is to explain how it works and how to configure the set-up.

During the installation of DB2, job DSNTIJSG creates several user defined functions. Some of these functions are the AMI interface from DB2 to WebSphere MQ.

To see which functions were created, you can issue the following SQL statement in SPUFI or QMF:

SELECT DISTINCT(NAME) FROM SYSIBM.SYSROUTINES WHERE SCHEMA = 'DB2MQ';

You should see the following:

MQREAD
MQREADALL
MQREADALLCLOB
MQREADCLOB
MQRECEIVE
MQRECEIVEALL
MQRECEIVEALLCLOB
MQRECEIVECLOB
MQSEND

In this post we will specifically be dealing with the MQSEND, MQREAD and MQRECEIVE user defined functions.

There are two tables that control the MQ Services within DB2:

SYSIBM.MQSERVICE_TABLE
SYSIBM.MQPOLICY_TABLE

The MQSERVICE_TABLE contains the service name, queue manager name, queue name, CCSID, encoding and a description of the service.

The MQPOLICY_TABLE contains items that are specific to the message itself and its delivery. You can think of the policy table as a mapping of a lot of the fields in the MQ Message Descriptor (MQMD). Items like, priority, persistence, expiry, message ID, correlation ID, COA, COD, reply to queue, reply to queue manager...and so on.

Let's define a service we can test with. Using SPUFI or QMF, execute the following:

INSERT INTO SYSIBM.MQSERVICE_TABLE VALUES('TEST_DB2_SEND','qmgrName','queueName',500,785,
'A TEST SERVICE');

Looking at the values, we have:

TEST_DB2_SEND - The service name
qmgrName - The name of the Queue Manager that you DB2 can access
queueName - The name of a local, remote or alias queue define on that queue manager
500 - Code character set ID (z/OS)
785 - Encoding (z/OS)
A TEST SERVICE - Description of the service

Now we will need a policy to govern the delivery of this message. For our example, we will keep it simple and specify that the message is to be persistent. So using SPUFI or QMF, execute

INSERT INTO SYSIBM.MQPOLICY_TABLE (POLICYNAME, SEND_PERSISTENCE, DESC)
VALUES('TEST_PERSIST_POLICY','Y','TEST POLICY FOR PERSISTENT MESSAGES');

In this statement, we specify the COLUMN names because there are a lot of columns on the table, and we only want these updated. The others will assume a default value. For the default values you can checkout the DB2 Information Center:

http://publib.boulder.ibm.com/infocenter/dzichelp/v2r2/index.jsp?topic=/com.ibm.db29.doc.apsg/db2z_db2mqtables.htm

So now we have a "service" as well as a "policy" defined, but before we can test the MQSEND function, we will need a Workload Manager (WLM) Application Environment established. Your z/OS systems programmer can set this up for you. Here are the values:

                     Application Environment
Name             DB21MQ
Description      TEST
Procedure Name   DB21MQ     Subsystem        DB2
Server Limit     No Limit
Start Parameters DB2SSN=&IWMSSNM,NUMTCB=20,APPLENV=DB21MQ

In the above example, we have a DB2 subsystem named DB21. Now we need a procedure in a system PROCLIB called DB21MQ..(see procedure name above).

DB21MQ   PROC RGN=0M,APPLENV=DB21MQ,DB2SSN=DB21,NUMTCB=20
IEFPROC EXEC PGM=DSNX9WLM,REGION=&RGN,TIME=NOLIMIT,
        PARM='&DB2SSN,&NUMTCB,&APPLENV'
STEPLIB DD DISP=SHR,DSN=db2hlq.SDSNEXIT
         DD DISP=SHR,DSN=db2hlq.SDSNLOAD
         DD DISP=SHR,DSN=mqhlq.SCSQLOAD
         DD DISP=SHR,DSN=mqhlq.SCSQAUTH
         DD DISP=SHR,DSN=mqhlq.SCSQANLE
DSSPRINT DD SYSOUT=*
SYSTSPRT DD SYSOUT=*
CEEDUMP DD SYSOUT=*
SYSPRINT DD SYSOUT=*
SYSABEND DD DUMMY

The db2hlq and mqhlq need to be replaced with your high level qualifier for your DB2 and MQ libraries. Okay..we should now be set to give this a test.

To test this we will using SPUFI and issuing a SQL statement to demonstrate the execution.

SELECT DB2MQ.MQSEND('TEST_DB2_SEND', 'TEST_PERSIST_POLICY',

'TEST MESSAGE WITH NEW POLICY')

FROM SYSIBM.SYSDUMMY1;

This SQL statement should have put the message, "TEST MESSAGE WITH NEW POLICY" on the queue you specified with the INSERT statement of the SYSIBM.MQSERVICE table.

Now let's define a service to read the queue. Using the same structure as before:

INSERT INTO SYSIBM.MQSERVICE_TABLE VALUES('TEST_DB2_READ','qmgrName','queueName',500,785,'A TEST READ SERVICE');

And now a policy. For this policy, we will just take the default values. You can tailor your policy to control your specific needs, but for this example..the defaults will work:

INSERT INTO SYSIBM.MQPOLICY_TABLE (POLICYNAME, DESC)
VALUES('TEST_READ_POLICY', 'POLICY FOR MQREAD');

We can now read the message from the queue using the following SQL statement:

SELECT DB2MQ.MQREAD('TEST_DB2_READ', 'TEST_READ_POLICY')
FROM SYSIBM.SYSDUMMY1;

Your output should display the contents of the message we placed on the queue. Using the MQREAD function, the message is still on the queue. To remove the message we will use the MQRECEIVE function.
So using the same policy as we did in the MQREAD...we can issue the statement:

SELECT DB2MQ.MQRECEIVE('TEST_DB2_READ', 'TEST_READ_POLICY')
FROM SYSIBM.SYSDUMMY1;

Your output should show the contents of the message, and the message is now removed from the queue.

You will probably want to name your services and policies something that makes sense based on the queue names and the specific behavior you are looking for. But this gives you an idea of what you are capable of doing. One practical use for these features would be to send and receive messages from a JDBC client without having the z/OS WebSphere MQ Client Attach Facility installed.

Keep messaging!!!!!

Sunday, February 27, 2011

WebSphere MQ V7 for z/OS and MQ Explorer....

One of the nice offerings in WebSphere MQ V7 on z/OS is the ability
for MQ Explorer to client attach without having to purchase the z/OS
Client Attachment Facility.

In version 7 on z/OS you are now allowed up to 5 client attachments
for use with MQ Explorer. This will allow administrators and enterprise
monitoring groups to view, monitor and alter z/OS Queue Managers using
the nice Windows interface that MQ Explorer offers.

To download the new version of MQ Explorer, go to: http://www01.ibm.com/software/integration/wmq/explorer/

The set-up for this is seemingly simple. On the z/OS side, you will need the
SYSTEM.ADMIN.SVRCONN channel defined. If you have the Client
Attach Facility installed, the MAXINST attribute on this channel can be set
between 0 and 999,999,999, without the Client Attach Facility the value for
MAXINST is limited to 5. The only other MQ object you will need is the SYSTEM.MQEXPLORER.REPLY.MODEL model queue.

Once these are set-up, your connection from MQ Explorer can be established. You will notice in thenew version of MQ Explorer there is a logon area for your z/OS ID and password. If you are the z/OS MQ administrator, you probably already have the necessary access to the queue manager and all of the objects, so no additional RACF (or ACF2, TopSecret) changes will be needed. However, if you are allowing others to use MQ Explorer who do not currently have access to the queue manager and objects, the following will be needed:

- RACF UPDATE authority to the system queue
SYSTEM.MQEXPLORER.REPLY.MODEL
- RACF UPDATE authority to the queues, AMQ.MQEXPLORER.*
- CONNECT authority to the target queue manager
- Correct authority for the action they specify. This can be different,
for different types of users.
- READ authority to all of the hlq.DISPLAY.object profiles in the MQCMDS
class

Enjoy!!