1 =============================================
2 Debugging on Linux for s/390 & z/Architecture
3 =============================================
5 Denis Joseph Barrow (djbarrow@de.ibm.com,barrow_dj@yahoo.com)
7 Copyright (C) 2000-2001 IBM Deutschland Entwicklung GmbH, IBM Corporation
9 .. Best viewed with fixed width fonts
13 This document is intended to give a good overview of how to debug Linux for
14 s/390 and z/Architecture. It is not intended as a complete reference and not a
15 tutorial on the fundamentals of C & assembly. It doesn't go into
16 390 IO in any detail. It is intended to complement the documents in the
17 reference section below & any other worthwhile references you get.
19 It is intended like the Enterprise Systems Architecture/390 Reference Summary
20 to be printed out & used as a quick cheat sheet self help style reference when
26 Address Spaces on Intel Linux
27 Address Spaces on Linux for s/390 & z/Architecture
28 The Linux for s/390 & z/Architecture Kernel Task Structure
29 Register Usage & Stackframes on Linux for s/390 & z/Architecture
30 A sample program with comments
31 Compiling programs for debugging on Linux for s/390 & z/Architecture
33 s/390 & z/Architecture IO Overview
34 Debugging IO on s/390 & z/Architecture under VM
35 GDB on s/390 & z/Architecture
36 Stack chaining in gdb by hand
47 The current architectures have the following registers.
49 16 General propose registers, 32 bit on s/390 and 64 bit on z/Architecture,
50 r0-r15 (or gpr0-gpr15), used for arithmetic and addressing.
52 16 Control registers, 32 bit on s/390 and 64 bit on z/Architecture, cr0-cr15,
53 kernel usage only, used for memory management, interrupt control, debugging
56 16 Access registers (ar0-ar15), 32 bit on both s/390 and z/Architecture,
57 normally not used by normal programs but potentially could be used as
58 temporary storage. These registers have a 1:1 association with general
59 purpose registers and are designed to be used in the so-called access
60 register mode to select different address spaces.
61 Access register 0 (and access register 1 on z/Architecture, which needs a
62 64 bit pointer) is currently used by the pthread library as a pointer to
63 the current running threads private area.
65 16 64-bit floating point registers (fp0-fp15 ) IEEE & HFP floating
66 point format compliant on G5 upwards & a Floating point control reg (FPC)
68 4 64-bit registers (fp0,fp2,fp4 & fp6) HFP only on older machines.
71 Linux (currently) always uses IEEE & emulates G5 IEEE format on older
72 machines, ( provided the kernel is configured for this ).
75 The PSW is the most important register on the machine it
76 is 64 bit on s/390 & 128 bit on z/Architecture & serves the roles of
77 a program counter (pc), condition code register,memory space designator.
78 In IBM standard notation I am counting bit 0 as the MSB.
79 It has several advantages over a normal program counter
80 in that you can change address translation & program counter
81 in a single instruction. To change address translation,
82 e.g. switching address translation off requires that you
83 have a logical=physical mapping for the address you are
86 +-------------------------+-------------------------------------------------+
88 +--------+----------------+ Value |
89 | s/390 | z/Architecture | |
90 +========+================+=================================================+
91 | 0 | 0 | Reserved (must be 0) otherwise specification |
92 | | | exception occurs. |
93 +--------+----------------+-------------------------------------------------+
94 | 1 | 1 | Program Event Recording 1 PER enabled, |
95 | | | PER is used to facilitate debugging e.g. |
96 | | | single stepping. |
97 +--------+----------------+-------------------------------------------------+
98 | 2-4 | 2-4 | Reserved (must be 0). |
99 +--------+----------------+-------------------------------------------------+
100 | 5 | 5 | Dynamic address translation 1=DAT on. |
101 +--------+----------------+-------------------------------------------------+
102 | 6 | 6 | Input/Output interrupt Mask |
103 +--------+----------------+-------------------------------------------------+
104 | 7 | 7 | External interrupt Mask used primarily for |
105 | | | interprocessor signalling and clock interrupts. |
106 +--------+----------------+-------------------------------------------------+
107 | 8-11 | 8-11 | PSW Key used for complex memory protection |
108 | | | mechanism (not used under linux) |
109 +--------+----------------+-------------------------------------------------+
110 | 12 | 12 | 1 on s/390 0 on z/Architecture |
111 +--------+----------------+-------------------------------------------------+
112 | 13 | 13 | Machine Check Mask 1=enable machine check |
114 +--------+----------------+-------------------------------------------------+
115 | 14 | 14 | Wait State. Set this to 1 to stop the processor |
116 | | | except for interrupts and give time to other |
117 | | | LPARS. Used in CPU idle in the kernel to |
118 | | | increase overall usage of processor resources. |
119 +--------+----------------+-------------------------------------------------+
120 | 15 | 15 | Problem state (if set to 1 certain instructions |
121 | | | are disabled). All linux user programs run with |
122 | | | this bit 1 (useful info for debugging under VM).|
123 +--------+----------------+-------------------------------------------------+
124 | 16-17 | 16-17 | Address Space Control |
126 | | | 00 Primary Space Mode: |
128 | | | The register CR1 contains the primary |
129 | | | address-space control element (PASCE), which |
130 | | | points to the primary space region/segment |
131 | | | table origin. |
133 | | | 01 Access register mode |
135 | | | 10 Secondary Space Mode: |
137 | | | The register CR7 contains the secondary |
138 | | | address-space control element (SASCE), which |
139 | | | points to the secondary space region or |
140 | | | segment table origin. |
142 | | | 11 Home Space Mode: |
144 | | | The register CR13 contains the home space |
145 | | | address-space control element (HASCE), which |
146 | | | points to the home space region/segment |
147 | | | table origin. |
149 | | | See "Address Spaces on Linux for s/390 & |
150 | | | z/Architecture" below for more information |
151 | | | about address space usage in Linux. |
152 +--------+----------------+-------------------------------------------------+
153 | 18-19 | 18-19 | Condition codes (CC) |
154 +--------+----------------+-------------------------------------------------+
155 | 20 | 20 | Fixed point overflow mask if 1=FPU exceptions |
156 | | | for this event occur (normally 0) |
157 +--------+----------------+-------------------------------------------------+
158 | 21 | 21 | Decimal overflow mask if 1=FPU exceptions for |
159 | | | this event occur (normally 0) |
160 +--------+----------------+-------------------------------------------------+
161 | 22 | 22 | Exponent underflow mask if 1=FPU exceptions |
162 | | | for this event occur (normally 0) |
163 +--------+----------------+-------------------------------------------------+
164 | 23 | 23 | Significance Mask if 1=FPU exceptions for this |
165 | | | event occur (normally 0) |
166 +--------+----------------+-------------------------------------------------+
167 | 24-31 | 24-30 | Reserved Must be 0. |
168 | +----------------+-------------------------------------------------+
169 | | 31 | Extended Addressing Mode |
170 | +----------------+-------------------------------------------------+
171 | | 32 | Basic Addressing Mode |
173 | | | Used to set addressing mode:: |
175 | | | +---------+----------+----------+ |
176 | | | | PSW 31 | PSW 32 | | |
177 | | | +---------+----------+----------+ |
178 | | | | 0 | 0 | 24 bit | |
179 | | | +---------+----------+----------+ |
180 | | | | 0 | 1 | 31 bit | |
181 | | | +---------+----------+----------+ |
182 | | | | 1 | 1 | 64 bit | |
183 | | | +---------+----------+----------+ |
184 +--------+----------------+-------------------------------------------------+
185 | 32 | | 1=31 bit addressing mode 0=24 bit addressing |
186 | | | mode (for backward compatibility), linux |
187 | | | always runs with this bit set to 1 |
188 +--------+----------------+-------------------------------------------------+
189 | 33-64 | | Instruction address. |
190 | +----------------+-------------------------------------------------+
191 | | 33-63 | Reserved must be 0 |
192 | +----------------+-------------------------------------------------+
193 | | 64-127 | Address |
195 | | | - In 24 bits mode bits 64-103=0 bits 104-127 |
197 | | | - In 31 bits mode bits 64-96=0 bits 97-127 |
201 | | | unlike 31 bit mode on s/390 bit 96 must be |
202 | | | zero when loading the address with LPSWE |
203 | | | otherwise a specification exception occurs, |
204 | | | LPSW is fully backward compatible. |
205 +--------+----------------+-------------------------------------------------+
209 This per cpu memory area is too intimately tied to the processor not to mention.
210 It exists between the real addresses 0-4096 on s/390 and between 0-8192 on
211 z/Architecture and is exchanged with one page on s/390 or two pages on
212 z/Architecture in absolute storage by the set prefix instruction during Linux
215 This page is mapped to a different prefix for each processor in an SMP
216 configuration (assuming the OS designer is sane of course).
218 Bytes 0-512 (200 hex) on s/390 and 0-512, 4096-4544, 4604-5119 currently on
219 z/Architecture are used by the processor itself for holding such information
220 as exception indications and entry points for exceptions.
222 Bytes after 0xc00 hex are used by linux for per processor globals on s/390 and
223 z/Architecture (there is a gap on z/Architecture currently between 0xc00 and
224 0x1000, too, which is used by Linux).
226 The closest thing to this on traditional architectures is the interrupt
227 vector table. This is a good thing & does simplify some of the kernel coding
228 however it means that we now cannot catch stray NULL pointers in the
229 kernel without hard coded checks.
233 Address Spaces on Intel Linux
234 =============================
236 The traditional Intel Linux is approximately mapped as follows forgive
239 0xFFFFFFFF 4GB Himem *****************
243 ***************** ****************
244 User Space Himem * User Stack * * *
245 (typically 0xC0000000 3GB ) ***************** * *
246 * Shared Libs * * Next Process *
247 ***************** * to *
253 0x00000000 ***************** ****************
255 Now it is easy to see that on Intel it is quite easy to recognise a kernel
256 address as being one greater than user space himem (in this case 0xC0000000),
257 and addresses of less than this are the ones in the current running program on
258 this processor (if an smp box).
260 If using the virtual machine ( VM ) as a debugger it is quite difficult to
261 know which user process is running as the address space you are looking at
262 could be from any process in the run queue.
264 The limitation of Intels addressing technique is that the linux
265 kernel uses a very simple real address to virtual addressing technique
266 of Real Address=Virtual Address-User Space Himem.
267 This means that on Intel the kernel linux can typically only address
268 Himem=0xFFFFFFFF-0xC0000000=1GB & this is all the RAM these machines
271 They can lower User Himem to 2GB or lower & thus be
272 able to use 2GB of RAM however this shrinks the maximum size
273 of User Space from 3GB to 2GB they have a no win limit of 4GB unless
277 On 390 our limitations & strengths make us slightly different.
278 For backward compatibility we are only allowed use 31 bits (2GB)
279 of our 32 bit addresses, however, we use entirely separate address
280 spaces for the user & kernel.
282 This means we can support 2GB of non Extended RAM on s/390, & more
283 with the Extended memory management swap device &
284 currently 4TB of physical memory currently on z/Architecture.
287 Address Spaces on Linux for s/390 & z/Architecture
288 ==================================================
290 Our addressing scheme is basically as follows::
292 Primary Space Home Space
293 Himem 0x7fffffff 2GB on s/390 ***************** ****************
294 currently 0x3ffffffffff (2^42)-1 * User Stack * * *
295 on z/Architecture. ***************** * *
297 ***************** * *
303 0x00000000 ***************** ****************
305 This also means that we need to look at the PSW problem state bit and the
306 addressing mode to decide whether we are looking at user or kernel space.
308 User space runs in primary address mode (or access register mode within
311 The kernel usually also runs in home space mode, however when accessing
312 user space the kernel switches to primary or secondary address mode if
313 the mvcos instruction is not available or if a compare-and-swap (futex)
314 instruction on a user space address is performed.
316 When also looking at the ASCE control registers, this means:
320 - runs in primary or access register mode
321 - cr1 contains the user asce
322 - cr7 contains the user asce
323 - cr13 contains the kernel asce
327 - runs in home space mode
328 - cr1 contains the user or kernel asce
330 - the kernel asce is loaded when a uaccess requires primary or
331 secondary address mode
333 - cr7 contains the user or kernel asce, (changed with set_fs())
334 - cr13 contains the kernel asce
336 In case of uaccess the kernel changes to:
338 - primary space mode in case of a uaccess (copy_to_user) and uses
339 e.g. the mvcp instruction to access user space. However the kernel
340 will stay in home space mode if the mvcos instruction is available
341 - secondary space mode in case of futex atomic operations, so that the
342 instructions come from primary address space and data from secondary
345 In case of KVM, the kernel runs in home space mode, but cr1 gets switched
346 to contain the gmap asce before the SIE instruction gets executed. When
347 the SIE instruction is finished, cr1 will be switched back to contain the
351 Virtual Addresses on s/390 & z/Architecture
352 ===========================================
354 A virtual address on s/390 is made up of 3 parts
355 The SX (segment index, roughly corresponding to the PGD & PMD in Linux
356 terminology) being bits 1-11.
358 The PX (page index, corresponding to the page table entry (pte) in Linux
359 terminology) being bits 12-19.
361 The remaining bits BX (the byte index are the offset in the page )
364 On z/Architecture in linux we currently make up an address from 4 parts.
366 - The region index bits (RX) 0-32 we currently use bits 22-32
367 - The segment index (SX) being bits 33-43
368 - The page index (PX) being bits 44-51
369 - The byte index (BX) being bits 52-63
372 1) s/390 has no PMD so the PMD is really the PGD also.
373 A lot of this stuff is defined in pgtable.h.
375 2) Also seeing as s/390's page indexes are only 1k in size
376 (bits 12-19 x 4 bytes per pte ) we use 1 ( page 4k )
377 to make the best use of memory by updating 4 segment indices
378 entries each time we mess with a PMD & use offsets
379 0,1024,2048 & 3072 in this page as for our segment indexes.
380 On z/Architecture our page indexes are now 2k in size
381 ( bits 12-19 x 8 bytes per pte ) we do a similar trick
382 but only mess with 2 segment indices each time we mess with
385 3) As z/Architecture supports up to a massive 5-level page table lookup we
386 can only use 3 currently on Linux ( as this is all the generic kernel
387 currently supports ) however this may change in future
388 this allows us to access ( according to my sums )
389 4TB of virtual storage per process i.e.
390 4096*512(PTES)*1024(PMDS)*2048(PGD) = 4398046511104 bytes,
391 enough for another 2 or 3 of years I think :-).
392 to do this we use a region-third-table designation type in
393 our address space control registers.
396 The Linux for s/390 & z/Architecture Kernel Task Structure
397 ==========================================================
398 Each process/thread under Linux for S390 has its own kernel task_struct
399 defined in linux/include/linux/sched.h
400 The S390 on initialisation & resuming of a process on a cpu sets
401 the __LC_KERNEL_STACK variable in the spare prefix area for this cpu
402 (which we use for per-processor globals).
404 The kernel stack pointer is intimately tied with the task structure for
405 each processor as follows::
408 ************************
409 * 1 page kernel stack *
411 ************************
412 * 1 page task_struct *
414 8K aligned ************************
417 ************************
418 * 2 page kernel stack *
420 ************************
421 * 2 page task_struct *
423 16K aligned ************************
425 What this means is that we don't need to dedicate any register or global
426 variable to point to the current running process & can retrieve it with the
427 following very simple construct for s/390 & one very similar for
430 static inline struct task_struct * get_current(void)
432 struct task_struct *current;
433 __asm__("lhi %0,-8192\n\t"
439 i.e. just anding the current kernel stack pointer with the mask -8192.
440 Thankfully because Linux doesn't have support for nested IO interrupts
441 & our devices have large buffers can survive interrupts being shut for
442 short amounts of time we don't need a separate stack for interrupts.
447 Register Usage & Stackframes on Linux for s/390 & z/Architecture
448 =================================================================
451 This is the code that gcc produces at the top & the bottom of
452 each function. It usually is fairly consistent & similar from
453 function to function & if you know its layout you can probably
454 make some headway in finding the ultimate cause of a problem
455 after a crash without a source level debugger.
457 Note: To follow stackframes requires a knowledge of C or Pascal &
458 limited knowledge of one assembly language.
460 It should be noted that there are some differences between the
461 s/390 and z/Architecture stack layouts as the z/Architecture stack layout
462 didn't have to maintain compatibility with older linkage formats.
467 This is a built in compiler function for runtime allocation
468 of extra space on the callers stack which is obviously freed
469 up on function exit ( e.g. the caller may choose to allocate nothing
470 of a buffer of 4k if required for temporary purposes ), it generates
471 very efficient code ( a few cycles ) when compared to alternatives
475 These are local variables on the stack, i.e they aren't in registers &
479 This is a pointer to the stack pointer before entering a
480 framed functions ( see frameless function ) prologue got by
481 dereferencing the address of the current stack pointer,
482 i.e. got by accessing the 32 bit value at the stack pointers
486 This is a pointer to the back of the literal pool which
487 is an area just behind each procedure used to store constants
491 The caller probably needs to save these registers if there
492 is something of value in them, on the stack or elsewhere before making a
493 call to another procedure so that it can restore it later.
496 The code generated by the compiler to return to the caller.
499 A frameless function in Linux for s390 & z/Architecture is one which doesn't
500 need more than the register save area (96 bytes on s/390, 160 on z/Architecture)
501 given to it by the caller.
503 A frameless function never:
505 1) Sets up a back chain.
507 3) Calls other normal functions
511 This is a pointer to the global-offset-table in ELF
512 ( Executable Linkable Format, Linux'es most common executable format ),
513 all globals & shared library objects are found using this pointer.
516 ELF shared libraries are typically only loaded when routines in the shared
517 library are actually first called at runtime. This is lazy binding.
519 procedure-linkage-table
520 This is a table found from the GOT which contains pointers to routines
521 in other shared libraries which can't be called to by easier means.
524 The code generated by the compiler to set up the stack frame.
527 This is extra area allocated on the stack of the calling function if the
528 parameters for the callee's cannot all be put in registers, the same
529 area can be reused by each function the caller calls.
532 A COFF executable format based concept of a procedure reference
533 actually being 8 bytes or more as opposed to a simple pointer to the routine.
534 This is typically defined as follows:
536 - Routine Descriptor offset 0=Pointer to Function
537 - Routine Descriptor offset 4=Pointer to Table of Contents
539 The table of contents/TOC is roughly equivalent to a GOT pointer.
540 & it means that shared libraries etc. can be shared between several
541 environments each with their own TOC.
544 This is used in nested functions a concept adopted from pascal
545 by gcc not used in ansi C or C++ ( although quite useful ), basically it
546 is a pointer used to reference local variables of enclosing functions.
547 You might come across this stuff once or twice in your lifetime.
551 The function below should return 11 though gcc may get upset & toss warnings
552 about unused variables::
566 s/390 & z/Architecture Register usage
567 =====================================
569 ======== ========================================== ===============
570 r0 used by syscalls/assembly call-clobbered
571 r1 used by syscalls/assembly call-clobbered
572 r2 argument 0 / return value 0 call-clobbered
573 r3 argument 1 / return value 1 (if long long) call-clobbered
574 r4 argument 2 call-clobbered
575 r5 argument 3 call-clobbered
577 r7 pointer-to arguments 5 to ... saved
580 r10 static-chain ( if nested function ) saved
581 r11 frame-pointer ( if function used alloca ) saved
582 r12 got-pointer saved
583 r13 base-pointer saved
584 r14 return-address saved
585 r15 stack-pointer saved
587 f0 argument 0 / return value ( float/double ) call-clobbered
588 f2 argument 1 call-clobbered
589 f4 z/Architecture argument 2 saved
590 f6 z/Architecture argument 3 saved
591 ======== ========================================== ===============
593 The remaining floating points
594 f1,f3,f5 f7-f15 are call-clobbered.
598 1) The only requirement is that registers which are used
599 by the callee are saved, e.g. the compiler is perfectly
600 capable of using r11 for purposes other than a frame a
601 frame pointer if a frame pointer is not needed.
602 2) In functions with variable arguments e.g. printf the calling procedure
603 is identical to one without variable arguments & the same number of
604 parameters. However, the prologue of this function is somewhat more
605 hairy owing to it having to move these parameters to the stack to
606 get va_start, va_arg & va_end to work.
607 3) Access registers are currently unused by gcc but are used in
608 the kernel. Possibilities exist to use them at the moment for
609 temporary storage but it isn't recommended.
610 4) Only 4 of the floating point registers are used for
611 parameter passing as older machines such as G3 only have only 4
612 & it keeps the stack frame compatible with other compilers.
613 However with IEEE floating point emulation under linux on the
614 older machines you are free to use the other 12.
615 5) A long long or double parameter cannot be have the
616 first 4 bytes in a register & the second four bytes in the
617 outgoing args area. It must be purely in the outgoing args
618 area if crossing this boundary.
619 6) Floating point parameters are mixed with outgoing args
620 on the outgoing args area in the order the are passed in as parameters.
621 7) Floating point arguments 2 & 3 are saved in the outgoing args area for
628 ========= ============== ======================================================
630 ========= ============== ======================================================
631 0 0 back chain ( a 0 here signifies end of back chain )
632 4 8 eos ( end of stack, not used on Linux for S390 used
633 in other linkage formats )
634 8 16 glue used in other s/390 linkage formats for saved
635 routine descriptors etc.
636 12 24 glue used in other s/390 linkage formats for saved
637 routine descriptors etc.
640 24 48 saved r6 of caller function
641 28 56 saved r7 of caller function
642 32 64 saved r8 of caller function
643 36 72 saved r9 of caller function
644 40 80 saved r10 of caller function
645 44 88 saved r11 of caller function
646 48 96 saved r12 of caller function
647 52 104 saved r13 of caller function
648 56 112 saved r14 of caller function
649 60 120 saved r15 of caller function
650 64 128 saved f4 of caller function
651 72 132 saved f6 of caller function
653 96 160 outgoing args passed from caller to callee
654 96+x 160+x possible stack alignment ( 8 bytes desirable )
655 96+x+y 160+x+y alloca space of caller ( if used )
656 96+x+y+z 160+x+y+z automatics of caller ( if used )
658 ========= ============== ======================================================
660 A sample program with comments.
661 ===============================
663 Comments on the function test
664 -----------------------------
665 1) It didn't need to set up a pointer to the constant pool gpr13 as it is not
667 2) This is a frameless function & no stack is bought.
668 3) The compiler was clever enough to recognise that it could return the
669 value in r2 as well as use it for the passed in parameter ( :-) ).
670 4) The basr ( branch relative & save ) trick works as follows the instruction
671 has a special case with r0,r0 with some instruction operands is understood as
672 the literal value 0, some risc architectures also do this ). So now
673 we are branching to the next address & the address new program counter is
674 in r13,so now we subtract the size of the function prologue we have executed
675 the size of the literal pool to get to the top of the literal pool::
678 0040037c int test(int b)
679 { # Function prologue below
680 40037c: 90 de f0 34 stm %r13,%r14,52(%r15) # Save registers r13 & r14
681 400380: 0d d0 basr %r13,%r0 # Set up pointer to constant pool using
682 400382: a7 da ff fa ahi %r13,-6 # basr trick
685 400386: a7 2a 00 05 ahi %r2,5 # add 5 to r2
687 # Function epilogue below
688 40038a: 98 de f0 34 lm %r13,%r14,52(%r15) # restore registers r13 & 14
689 40038e: 07 fe br %r14 # return
692 Comments on the function main
693 -----------------------------
694 1) The compiler did this function optimally ( 8-) )::
696 Literal pool for main.
697 400390: ff ff ff ec .long 0xffffffec
698 main(int argc,char *argv[])
699 { # Function prologue below
700 400394: 90 bf f0 2c stm %r11,%r15,44(%r15) # Save necessary registers
701 400398: 18 0f lr %r0,%r15 # copy stack pointer to r0
702 40039a: a7 fa ff a0 ahi %r15,-96 # Make area for callee saving
703 40039e: 0d d0 basr %r13,%r0 # Set up r13 to point to
704 4003a0: a7 da ff f0 ahi %r13,-16 # literal pool
705 4003a4: 50 00 f0 00 st %r0,0(%r15) # Save backchain
707 return(test(5)); # Main Program Below
708 4003a8: 58 e0 d0 00 l %r14,0(%r13) # load relative address of test from
710 4003ac: a7 28 00 05 lhi %r2,5 # Set first parameter to 5
711 4003b0: 4d ee d0 00 bas %r14,0(%r14,%r13) # jump to test setting r14 as return
712 # address using branch & save instruction.
714 # Function Epilogue below
715 4003b4: 98 bf f0 8c lm %r11,%r15,140(%r15)# Restore necessary registers.
716 4003b8: 07 fe br %r14 # return to do program exit
725 main(int argc,char *argv[])
727 4004fc: 90 7f f0 1c stm %r7,%r15,28(%r15)
728 400500: a7 d5 00 04 bras %r13,400508 <main+0xc>
729 400504: 00 40 04 f4 .long 0x004004f4
730 # compiler now puts constant pool in code to so it saves an instruction
731 400508: 18 0f lr %r0,%r15
732 40050a: a7 fa ff a0 ahi %r15,-96
733 40050e: 50 00 f0 00 st %r0,0(%r15)
735 400512: 58 10 d0 00 l %r1,0(%r13)
736 400516: a7 28 00 05 lhi %r2,5
737 40051a: 0d e1 basr %r14,%r1
738 # compiler adds 1 extra instruction to epilogue this is done to
739 # avoid processor pipeline stalls owing to data dependencies on g5 &
740 # above as register 14 in the old code was needed directly after being loaded
741 # by the lm %r11,%r15,140(%r15) for the br %14.
742 40051c: 58 40 f0 98 l %r4,152(%r15)
743 400520: 98 7f f0 7c lm %r7,%r15,124(%r15)
748 Hartmut ( our compiler developer ) also has been threatening to take out the
749 stack backchain in optimised code as this also causes pipeline stalls, you
752 64 bit z/Architecture code disassembly
753 --------------------------------------
755 If you understand the stuff above you'll understand the stuff
756 below too so I'll avoid repeating myself & just say that
757 some of the instructions have g's on the end of them to indicate
758 they are 64 bit & the stack offsets are a bigger,
759 the only other difference you'll find between 32 & 64 bit is that
760 we now use f4 & f6 for floating point arguments on 64 bit::
762 00000000800005b0 <test>:
766 800005b0: a7 2a 00 05 ahi %r2,5
767 800005b4: b9 14 00 22 lgfr %r2,%r2 # downcast to integer
768 800005b8: 07 fe br %r14
769 800005ba: 07 07 bcr 0,%r7
774 00000000800005bc <main>:
775 main(int argc,char *argv[])
777 800005bc: eb bf f0 58 00 24 stmg %r11,%r15,88(%r15)
778 800005c2: b9 04 00 1f lgr %r1,%r15
779 800005c6: a7 fb ff 60 aghi %r15,-160
780 800005ca: e3 10 f0 00 00 24 stg %r1,0(%r15)
782 800005d0: a7 29 00 05 lghi %r2,5
783 # brasl allows jumps > 64k & is overkill here bras would do fune
784 800005d4: c0 e5 ff ff ff ee brasl %r14,800005b0 <test>
785 800005da: e3 40 f1 10 00 04 lg %r4,272(%r15)
786 800005e0: eb bf f0 f8 00 04 lmg %r11,%r15,248(%r15)
787 800005e6: 07 f4 br %r4
792 Compiling programs for debugging on Linux for s/390 & z/Architecture
793 ====================================================================
794 -gdwarf-2 now works it should be considered the default debugging
795 format for s/390 & z/Architecture as it is more reliable for debugging
796 shared libraries, normal -g debugging works much better now
797 Thanks to the IBM java compiler developers bug reports.
799 This is typically done adding/appending the flags -g or -gdwarf-2 to the
800 CFLAGS & LDFLAGS variables Makefile of the program concerned.
802 If using gdb & you would like accurate displays of registers &
803 stack traces compile without optimisation i.e make sure
804 that there is no -O2 or similar on the CFLAGS line of the Makefile &
805 the emitted gcc commands, obviously this will produce worse code
806 ( not advisable for shipment ) but it is an aid to the debugging process.
808 This aids debugging because the compiler will copy parameters passed in
809 in registers onto the stack so backtracing & looking at passed in
810 parameters will work, however some larger programs which use inline functions
811 will not compile without optimisation.
813 Debugging with optimisation has since much improved after fixing
814 some bugs, please make sure you are using gdb-5.0 or later developed
824 Addresses & values in the VM debugger are always hex never decimal
825 Address ranges are of the format <HexValue1>-<HexValue2> or
826 <HexValue1>.<HexValue2>
827 For example, the address range 0x2000 to 0x3000 can be described as 2000-3000
830 The VM Debugger is case insensitive.
832 VM's strengths are usually other debuggers weaknesses you can get at any
833 resource no matter how sensitive e.g. memory management resources, change
834 address translation in the PSW. For kernel hacking you will reap dividends if
837 The VM Debugger displays operators but not operands, and also the debugger
838 displays useful information on the same line as the author of the code probably
839 felt that it was a good idea not to go over the 80 columns on the screen.
840 This isn't as unintuitive as it may seem as the s/390 instructions are easy to
841 decode mentally and you can make a good guess at a lot of them as all the
842 operands are nibble (half byte aligned).
843 So if you have an objdump listing by hand, it is quite easy to follow, and if
844 you don't have an objdump listing keep a copy of the s/390 Reference Summary
845 or alternatively the s/390 principles of operation next to you.
846 e.g. even I can guess that
847 0001AFF8' LR 180F CC 0
848 is a ( load register ) lr r0,r15
850 Also it is very easy to tell the length of a 390 instruction from the 2 most
851 significant bits in the instruction (not that this info is really useful except
852 if you are trying to make sense of a hexdump of code).
855 ======================= ==================
856 Bits Instruction Length
857 ======================= ==================
862 ======================= ==================
864 The debugger also displays other useful info on the same line such as the
865 addresses being operated on destination addresses of branches & condition codes.
868 00019736' AHI A7DAFF0E CC 1
869 000198BA' BRC A7840004 -> 000198C2' CC 0
870 000198CE' STM 900EF068 >> 0FA95E78 CC 2
874 Useful VM debugger commands
875 ---------------------------
877 I suppose I'd better mention this before I start
878 to list the current active traces do::
882 there can be a maximum of 255 of these per set
883 ( more about trace sets later ).
885 To stop traces issue a::
889 To delete a particular breakpoint issue::
891 TR DEL <breakpoint number>
893 The PA1 key drops to CP mode so you can issue debugger commands,
894 Doing alt c (on my 3270 console at least ) clears the screen.
896 hitting b <enter> comes back to the running operating system
897 from cp mode ( in our case linux ).
899 It is typically useful to add shortcuts to your profile.exec file
900 if you have one ( this is roughly equivalent to autoexec.bat in DOS ).
901 file here are a few from mine::
903 /* this gives me command history on issuing f12 */
907 /* goes to trace set a */
908 set pf1 imm tr goto a
909 /* goes to trace set b */
910 set pf2 imm tr goto b
911 /* goes to trace set c */
912 set pf3 imm tr goto c
918 Setting a simple breakpoint::
922 To debug a particular function try::
924 TR I R <function address range>
925 TR I on its own will single step.
926 TR I DATA <MNEMONIC> <OPTIONAL RANGE> will trace for particular mnemonics
930 TR I DATA 4D R 0197BC.4000
932 will trace for BAS'es ( opcode 4D ) in the range 0197BC.4000
934 if you were inclined you could add traces for all branch instructions &
935 suffix them with the run prefix so you would have a backtrace on screen
936 when a program crashes::
938 TR BR <INTO OR FROM> will trace branches into or out of an address.
944 is often quite useful if a program is getting awkward & deciding
945 to branch to 0 & crashing as this will stop at the address before in jumps to 0.
949 TR I R <address range> RUN cmd d g
951 single steps a range of addresses but stays running &
952 displays the gprs on each step.
956 Displaying & modifying Registers
957 --------------------------------
959 will display all the gprs
961 Adding a extra G to all the commands is necessary to access the full 64 bit
962 content in VM on z/Architecture. Obviously this isn't required for access
963 registers as these are still 32 bit.
971 will display all the control registers
973 will display all the access registers
975 will display access registers 4 to 7
977 will display the GRPS of all CPUS in the configuration
979 will display the current PSW
981 will put the value 2000 into the PSW & cause crash your machine.
983 displays the prefix offset
988 To display memory mapped using the current PSW's mapping try::
992 To make VM display a message each time it hits a particular address and
996 will disassemble/display a range of instructions.
999 will store a 32 bit aligned address
1001 will display the EBCDIC in an address (if you are that way inclined)
1003 will display real addresses ( without DAT ) but with prefixing.
1005 There are other complex options to display if you need to get at say home space
1006 but are in primary space the easiest thing to do is to temporarily
1007 modify the PSW to the other addressing mode, display the stuff & then
1014 If you want to issue a debugger command without halting your virtual machine
1015 with the PA1 key try prefixing the command with #CP e.g.::
1019 also suffixing most debugger commands with RUN will cause them not
1020 to stop just display the mnemonic at the current instruction on the console.
1022 If you have several breakpoints you want to put into your program &
1023 you get fed up of cross referencing with System.map
1024 you can do the following trick for several symbols.
1028 grep do_signal System.map
1030 which emits the following among other things::
1032 0001f4e0 T do_signal
1036 TR I PSWA 0001f4e0 cmd msg * do_signal
1038 This sends a message to your own console each time do_signal is entered.
1039 ( As an aside I wrote a perl script once which automatically generated a REXX
1040 script with breakpoints on every kernel procedure, this isn't a good idea
1041 because there are thousands of these routines & VM can only set 255 breakpoints
1042 at a time so you nearly had to spend as long pruning the file down as you would
1043 entering the msgs by hand), however, the trick might be useful for a single
1044 object file. In the 3270 terminal emulator x3270 there is a very useful option
1045 in the file menu called "Save Screen In File" - this is very good for keeping a
1048 From CMS help <command name> will give you online help on a particular command.
1053 Also CP has a file called profile.exec which automatically gets called
1054 on startup of CMS ( like autoexec.bat ), keeping on a DOS analogy session
1055 CP has a feature similar to doskey, it may be useful for you to
1056 use profile.exec to define some keystrokes.
1059 This does a single step in VM on pressing F8.
1062 This sets up the ^ key.
1063 which can be used for ^c (ctrl-c),^z (ctrl-z) which can't be typed
1064 directly into some 3270 consoles.
1067 This types the starting keystrokes for a sysrq see SysRq below.
1069 This retrieves command history on pressing F12.
1072 Sometimes in VM the display is set up to scroll automatically this
1073 can be very annoying if there are messages you wish to look at
1077 This will nearly stop automatic screen updates, however it will
1078 cause a denial of service if lots of messages go to the 3270 console,
1079 so it would be foolish to use this as the default on a production machine.
1082 Tracing particular processes
1083 ----------------------------
1084 The kernel's text segment is intentionally at an address in memory that it will
1085 very seldom collide with text segments of user programs ( thanks Martin ),
1086 this simplifies debugging the kernel.
1087 However it is quite common for user processes to have addresses which collide
1088 this can make debugging a particular process under VM painful under normal
1089 circumstances as the process may change when doing a::
1091 TR I R <address range>.
1093 Thankfully after reading VM's online help I figured out how to debug
1094 I particular process.
1096 Your first problem is to find the STD ( segment table designation )
1097 of the program you wish to debug.
1098 There are several ways you can do this here are a few
1102 objdump --syms <program to be debugged> | grep main
1104 To get the address of main in the program. Then::
1106 tr i pswa <address of main>
1108 Start the program, if VM drops to CP on what looks like the entry
1109 point of the main function this is most likely the process you wish to debug.
1110 Now do a D X13 or D XG13 on z/Architecture.
1112 On 31 bit the STD is bits 1-19 ( the STO segment table origin )
1113 & 25-31 ( the STL segment table length ) of CR13.
1117 TR I R STD <CR13's value> 0.7fffffff
1121 TR I R STD 8F32E1FF 0.7fffffff
1123 Another very useful variation is::
1125 TR STORE INTO STD <CR13's value> <address range>
1127 for finding out when a particular variable changes.
1129 An alternative way of finding the STD of a currently running process
1130 is to do the following, ( this method is more complex but
1131 could be quite convenient if you aren't updating the kernel much &
1132 so your kernel structures will stay constant for a reasonable period of
1137 grep task /proc/<pid>/status
1139 from this you should see something like::
1141 task: 0f160000 ksp: 0f161de8 pt_regs: 0f161f68
1143 This now gives you a pointer to the task structure.
1147 CC:="s390-gcc -g" kernel/sched.s
1149 To get the task_struct stabinfo.
1151 ( task_struct is defined in include/linux/sched.h ).
1153 Now we want to look at
1154 task->active_mm->pgd
1156 on my machine the active_mm in the task structure stab is
1157 active_mm:(4,12),672,32
1159 its offset is 672/8=84=0x54
1161 the pgd member in the mm_struct stab is
1162 pgd:(4,6)=*(29,5),96,32
1163 so its offset is 96/8=12=0xc
1167 hexdump -s 0xf160054 /dev/mem | more
1169 i.e. task_struct+active_mm offset
1170 to look at the active_mm member::
1172 f160054 0fee cc60 0019 e334 0000 0000 0000 0011
1176 hexdump -s 0x0feecc6c /dev/mem | more
1178 i.e. active_mm+pgd offset::
1180 feecc6c 0f2c 0000 0000 0001 0000 0001 0000 0010
1182 we get something like
1185 TR I R STD <pgd|0x7f> 0.7fffffff
1187 i.e. the 0x7f is added because the pgd only
1188 gives the page table origin & we need to set the low bits
1189 to the maximum possible segment table length.
1193 TR I R STD 0f2c007f 0.7fffffff
1195 on z/Architecture you'll probably need to do::
1197 TR I R STD <pgd|0x7> 0.ffffffffffffffff
1199 to set the TableType to 0x1 & the Table length to 3.
1203 Tracing Program Exceptions
1204 --------------------------
1205 If you get a crash which says something like
1206 illegal operation or specification exception followed by a register dump
1207 You can restart linux & trace these using the tr prog <range or value> trace
1211 The most common ones you will normally be tracing for is:
1213 - 1=operation exception
1214 - 2=privileged operation exception
1215 - 4=protection exception
1216 - 5=addressing exception
1217 - 6=specification exception
1218 - 10=segment translation exception
1219 - 11=page translation exception
1221 The full list of these is on page 22 of the current s/390 Reference Summary.
1224 tr prog 10 will trace segment translation exceptions.
1226 tr prog on its own will trace all program interruption codes.
1230 On starting VM you are initially in the INITIAL trace set.
1231 You can do a Q TR to verify this.
1232 If you have a complex tracing situation where you wish to wait for instance
1233 till a driver is open before you start tracing IO, but know in your
1234 heart that you are going to have to make several runs through the code till you
1235 have a clue whats going on.
1237 What you can do is::
1239 TR I PSWA <Driver open address>
1241 hit b to continue till breakpoint
1243 reach the breakpoint
1248 TR IO 7c08-7c09 inst int run
1250 or whatever the IO channels you wish to trace are & hit b
1252 To got back to the initial trace set do::
1256 & the TR I PSWA <Driver open address> will be the only active breakpoint again.
1259 Tracing linux syscalls under VM
1260 -------------------------------
1261 Syscalls are implemented on Linux for S390 by the Supervisor call instruction
1262 (SVC). There 256 possibilities of these as the instruction is made up of a 0xA
1263 opcode and the second byte being the syscall number. They are traced using the
1266 TR SVC <Optional value or range>
1268 the syscalls are defined in linux/arch/s390/include/asm/unistd.h
1269 e.g. to trace all file opens just do::
1271 TR SVC 5 ( as this is the syscall number of open )
1274 SMP Specific commands
1275 ---------------------
1276 To find out how many cpus you have
1277 Q CPUS displays all the CPU's available to your virtual machine
1278 To find the cpu that the current cpu VM debugger commands are being directed at
1279 do Q CPU to change the current cpu VM debugger commands are being directed at
1282 CPU <desired cpu no>
1284 On a SMP guest issue a command to all CPUs try prefixing the command with cpu
1285 all. To issue a command to a particular cpu try cpu <cpu number> e.g.::
1287 CPU 01 TR I R 2000.3000
1289 If you are running on a guest with several cpus & you have a IO related problem
1290 & cannot follow the flow of code but you know it isn't smp related.
1292 from the bash prompt issue::
1294 shutdown -h now or halt.
1300 to find out how many cpus you have detach each one of them from cp except
1301 cpu 0 by issuing a::
1303 DETACH CPU 01-(number of cpus in configuration)
1308 will trace inter processor signal processor instructions.
1310 DEFINE CPU 01-(number in configuration)
1311 will get your guests cpus back.
1314 Help for displaying ascii textstrings
1315 -------------------------------------
1316 On the very latest VM Nucleus'es VM can now display ascii
1317 ( thanks Neale for the hint ) by doing::
1327 Under older VM debuggers (I love EBDIC too) you can use following little
1328 program which converts a command line of hex digits to ascii text. It can be
1329 compiled under linux and you can copy the hex digits from your x3270 terminal
1330 to your xterm if you are debugging from a linuxbox.
1332 This is quite useful when looking at a parameter passed in as a text string
1333 under VM ( unless you are good at decoding ASCII in your head ).
1335 e.g. consider tracing an open syscall::
1339 We have stopped at a breakpoint::
1341 000151B0' SVC 0A05 -> 0001909A' CC 0
1343 D 20.8 to check the SVC old psw in the prefix area and see was it from userspace
1344 (for the layout of the prefix area consult the "Fixed Storage Locations"
1345 chapter of the s/390 Reference Summary if you have it available).
1349 V00000020 070C2000 800151B2
1351 The problem state bit wasn't set & it's also too early in the boot sequence
1352 for it to be a userspace SVC if it was we would have to temporarily switch the
1353 psw to user space addressing so we could get at the first parameter of the open
1361 Now display what gpr2 is pointing to::
1364 V00014CB4 2F646576 2F636F6E 736F6C65 00001BF5
1365 V00014CC4 FC00014C B4001001 E0001000 B8070707
1367 Now copy the text till the first 00 hex ( which is the end of the string
1368 to an xterm & do hex2ascii on it::
1370 hex2ascii 2F646576 2F636F6E 736F6C65 00
1374 Decoded Hex:=/ d e v / c o n s o l e 0x00
1376 We were opening the console device,
1378 You can compile the code below yourself for practice :-),
1384 * a useful little tool for converting a hexadecimal command line to ascii
1386 * Author(s): Denis Joseph Barrow (djbarrow@de.ibm.com,barrow_dj@yahoo.com)
1387 * (C) 2000 IBM Deutschland Entwicklung GmbH, IBM Corporation.
1391 int main(int argc,char *argv[])
1393 int cnt1,cnt2,len,toggle=0;
1395 unsigned char c,hex;
1397 if(argc>1&&(strcmp(argv[1],"-a")==0))
1399 printf("Decoded Hex:=");
1400 for(cnt1=startcnt;cnt1<argc;cnt1++)
1402 len=strlen(argv[cnt1]);
1403 for(cnt2=0;cnt2<len;cnt2++)
1423 printf("0x%02X ",(int)hex);
1444 Stack tracing under VM
1445 ----------------------
1449 Here are the tricks I use 9 out of 10 times it works pretty well,
1451 When your backchain reaches a dead end
1452 --------------------------------------
1453 This can happen when an exception happens in the kernel and the kernel is
1454 entered twice. If you reach the NULL pointer at the end of the back chain you
1455 should be able to sniff further back if you follow the following tricks.
1456 1) A kernel address should be easy to recognise since it is in
1457 primary space & the problem state bit isn't set & also
1458 The Hi bit of the address is set.
1459 2) Another backchain should also be easy to recognise since it is an
1460 address pointing to another address approximately 100 bytes or 0x70 hex
1461 behind the current stackpointer.
1464 Here is some practice.
1466 boot the kernel & hit PA1 at some random time
1468 d g to display the gprs, this should display something like::
1470 GPR 0 = 00000001 00156018 0014359C 00000000
1471 GPR 4 = 00000001 001B8888 000003E0 00000000
1472 GPR 8 = 00100080 00100084 00000000 000FE000
1473 GPR 12 = 00010400 8001B2DC 8001B36A 000FFED8
1475 Note that GPR14 is a return address but as we are real men we are going to
1477 display 0x40 bytes after the stack pointer::
1479 V000FFED8 000FFF38 8001B838 80014C8E 000FFF38
1480 V000FFEE8 00000000 00000000 000003E0 00000000
1481 V000FFEF8 00100080 00100084 00000000 000FE000
1482 V000FFF08 00010400 8001B2DC 8001B36A 000FFED8
1485 Ah now look at whats in sp+56 (sp+0x38) this is 8001B36A our saved r14 if
1486 you look above at our stackframe & also agrees with GPR14.
1492 we now are taking the contents of SP to get our first backchain::
1494 V000FFF38 000FFFA0 00000000 00014995 00147094
1495 V000FFF48 00147090 001470A0 000003E0 00000000
1496 V000FFF58 00100080 00100084 00000000 001BF1D0
1497 V000FFF68 00010400 800149BA 80014CA6 000FFF38
1499 This displays a 2nd return address of 80014CA6
1505 for our 3rd backchain::
1507 V000FFFA0 04B52002 0001107F 00000000 00000000
1508 V000FFFB0 00000000 00000000 FF000000 0001107F
1509 V000FFFC0 00000000 00000000 00000000 00000000
1510 V000FFFD0 00010400 80010802 8001085A 000FFFA0
1513 our 3rd return address is 8001085A
1515 as the 04B52002 looks suspiciously like rubbish it is fair to assume that the
1516 kernel entry routines for the sake of optimisation don't set up a backchain.
1518 now look at System.map to see if the addresses make any sense::
1520 grep -i 0001b3 System.map
1522 outputs among other things::
1527 is cpu_idle+0x66 ( quiet the cpu is asleep, don't wake it )
1531 grep -i 00014 System.map
1533 produces among other things::
1535 00014a78 T start_kernel
1537 so 0014CA6 is start_kernel+some hex number I can't add in my head.
1541 grep -i 00108 System.map
1547 so 8001085A is _stext+0x5a
1549 Congrats you've done your first backchain.
1553 s/390 & z/Architecture IO Overview
1554 ==================================
1556 I am not going to give a course in 390 IO architecture as this would take me
1557 quite a while and I'm no expert. Instead I'll give a 390 IO architecture
1558 summary for Dummies. If you have the s/390 principles of operation available
1559 read this instead. If nothing else you may find a few useful keywords in here
1560 and be able to use them on a web search engine to find more useful information.
1562 Unlike other bus architectures modern 390 systems do their IO using mostly
1563 fibre optics and devices such as tapes and disks can be shared between several
1564 mainframes. Also S390 can support up to 65536 devices while a high end PC based
1565 system might be choking with around 64.
1567 Here is some of the common IO terminology:
1570 This is the logical number most IO commands use to talk to an IO device. There
1571 can be up to 0x10000 (65536) of these in a configuration, typically there are a
1572 few hundred. Under VM for simplicity they are allocated contiguously, however
1573 on the native hardware they are not. They typically stay consistent between
1574 boots provided no new hardware is inserted or removed.
1576 Under Linux for s390 we use these as IRQ's and also when issuing an IO command
1577 (CLEAR SUBCHANNEL, HALT SUBCHANNEL, MODIFY SUBCHANNEL, RESUME SUBCHANNEL,
1578 START SUBCHANNEL, STORE SUBCHANNEL and TEST SUBCHANNEL). We use this as the ID
1579 of the device we wish to talk to. The most important of these instructions are
1580 START SUBCHANNEL (to start IO), TEST SUBCHANNEL (to check whether the IO
1581 completed successfully) and HALT SUBCHANNEL (to kill IO). A subchannel can have
1582 up to 8 channel paths to a device, this offers redundancy if one is not
1586 This number remains static and is closely tied to the hardware. There are 65536
1587 of these, made up of a CHPID (Channel Path ID, the most significant 8 bits) and
1588 another lsb 8 bits. These remain static even if more devices are inserted or
1589 removed from the hardware. There is a 1 to 1 mapping between subchannels and
1590 device numbers, provided devices aren't inserted or removed.
1592 Channel Control Words:
1593 CCWs are linked lists of instructions initially pointed to by an operation
1594 request block (ORB), which is initially given to Start Subchannel (SSCH)
1595 command along with the subchannel number for the IO subsystem to process
1596 while the CPU continues executing normal code.
1597 CCWs come in two flavours, Format 0 (24 bit for backward compatibility) and
1598 Format 1 (31 bit). These are typically used to issue read and write (and many
1599 other) instructions. They consist of a length field and an absolute address
1602 Each IO typically gets 1 or 2 interrupts, one for channel end (primary status)
1603 when the channel is idle, and the second for device end (secondary status).
1604 Sometimes you get both concurrently. You check how the IO went on by issuing a
1605 TEST SUBCHANNEL at each interrupt, from which you receive an Interruption
1606 response block (IRB). If you get channel and device end status in the IRB
1607 without channel checks etc. your IO probably went okay. If you didn't you
1608 probably need to examine the IRB, extended status word etc.
1609 If an error occurs, more sophisticated control units have a facility known as
1610 concurrent sense. This means that if an error occurs Extended sense information
1611 will be presented in the Extended status word in the IRB. If not you have to
1612 issue a subsequent SENSE CCW command after the test subchannel.
1615 TPI (Test pending interrupt) can also be used for polled IO, but in
1616 multitasking multiprocessor systems it isn't recommended except for
1617 checking special cases (i.e. non looping checks for pending IO etc.).
1619 Store Subchannel and Modify Subchannel can be used to examine and modify
1620 operating characteristics of a subchannel (e.g. channel paths).
1622 Other IO related Terms:
1625 S390's Clustering Technology
1627 S390's new high speed IO architecture to support devices such as gigabit
1628 ethernet, this architecture is also designed to be forward compatible with
1629 upcoming 64 bit machines.
1635 Input Output Processors (IOP's) are responsible for communicating between
1636 the mainframe CPU's & the channel & relieve the mainframe CPU's from the
1637 burden of communicating with IO devices directly, this allows the CPU's to
1638 concentrate on data processing.
1640 IOP's can use one or more links ( known as channel paths ) to talk to each
1641 IO device. It first checks for path availability & chooses an available one,
1642 then starts ( & sometimes terminates IO ).
1643 There are two types of channel path: ESCON & the Parallel IO interface.
1645 IO devices are attached to control units, control units provide the
1646 logic to interface the channel paths & channel path IO protocols to
1647 the IO devices, they can be integrated with the devices or housed separately
1648 & often talk to several similar devices ( typical examples would be raid
1649 controllers or a control unit which connects to 1000 3270 terminals )::
1652 +---------------------------------------------------------------+
1653 | +-----+ +-----+ +-----+ +-----+ +----------+ +----------+ |
1654 | | CPU | | CPU | | CPU | | CPU | | Main | | Expanded | |
1655 | | | | | | | | | | Memory | | Storage | |
1656 | +-----+ +-----+ +-----+ +-----+ +----------+ +----------+ |
1657 |---------------------------------------------------------------+
1659 |---------------------------------------------------------------
1660 | C | C | C | C | C | C | C | C | C | C | C | C | C | C | C | C |
1661 ----------------------------------------------------------------
1663 || Bus & Tag Channel Path || ESCON
1664 || ====================== || Channel
1666 +----------+ +----------+ +----------+
1668 | CU | | CU | | CU |
1670 +----------+ +----------+ +----------+
1672 +----------+ +----------+ +----------+ +----------+ +----------+
1673 |I/O Device| |I/O Device| |I/O Device| |I/O Device| |I/O Device|
1674 +----------+ +----------+ +----------+ +----------+ +----------+
1675 CPU = Central Processing Unit
1680 The 390 IO systems come in 2 flavours the current 390 machines support both
1682 The Older 360 & 370 Interface,sometimes called the Parallel I/O interface,
1683 sometimes called Bus-and Tag & sometimes Original Equipment Manufacturers
1686 This byte wide Parallel channel path/bus has parity & data on the "Bus" cable
1687 and control lines on the "Tag" cable. These can operate in byte multiplex mode
1688 for sharing between several slow devices or burst mode and monopolize the
1689 channel for the whole burst. Up to 256 devices can be addressed on one of these
1690 cables. These cables are about one inch in diameter. The maximum unextended
1691 length supported by these cables is 125 Meters but this can be extended up to
1692 2km with a fibre optic channel extended such as a 3044. The maximum burst speed
1693 supported is 4.5 megabytes per second. However, some really old processors
1694 support only transfer rates of 3.0, 2.0 & 1.0 MB/sec.
1695 One of these paths can be daisy chained to up to 8 control units.
1698 ESCON if fibre optic it is also called FICON
1699 Was introduced by IBM in 1990. Has 2 fibre optic cables and uses either leds or
1700 lasers for communication at a signaling rate of up to 200 megabits/sec. As
1701 10bits are transferred for every 8 bits info this drops to 160 megabits/sec
1702 and to 18.6 Megabytes/sec once control info and CRC are added. ESCON only
1703 operates in burst mode.
1705 ESCONs typical max cable length is 3km for the led version and 20km for the
1706 laser version known as XDF (extended distance facility). This can be further
1707 extended by using an ESCON director which triples the above mentioned ranges.
1708 Unlike Bus & Tag as ESCON is serial it uses a packet switching architecture,
1709 the standard Bus & Tag control protocol is however present within the packets.
1710 Up to 256 devices can be attached to each control unit that uses one of these
1713 Common 390 Devices include:
1714 Network adapters typically OSA2,3172's,2116's & OSA-E gigabit ethernet adapters,
1715 Consoles 3270 & 3215 (a teletype emulated under linux for a line mode console).
1716 DASD's direct access storage devices ( otherwise known as hard disks ).
1718 CTC ( Channel to Channel Adapters ),
1719 ESCON or Parallel Cables used as a very high speed serial link
1723 Debugging IO on s/390 & z/Architecture under VM
1724 ===============================================
1726 Now we are ready to go on with IO tracing commands under VM
1728 A few self explanatory queries::
1732 Q DISK ( This command is CMS specific )
1735 Q OSA on my machine returns::
1737 OSA 7C08 ON OSA 7C08 SUBCHANNEL = 0000
1738 OSA 7C09 ON OSA 7C09 SUBCHANNEL = 0001
1739 OSA 7C14 ON OSA 7C14 SUBCHANNEL = 0002
1740 OSA 7C15 ON OSA 7C15 SUBCHANNEL = 0003
1742 If you have a guest with certain privileges you may be able to see devices
1743 which don't belong to you. To avoid this, add the option V.
1748 Now using the device numbers returned by this command we will
1749 Trace the io starting up on the first device 7c08 & 7c09
1750 In our simplest case we can trace the
1752 like TR SSCH 7C08-7C09
1753 or the halt subchannels
1754 or TR HSCH 7C08-7C09
1755 MSCH's ,STSCH's I think you can guess the rest
1757 A good trick is tracing all the IO's and CCWS and spooling them into the reader
1758 of another VM guest so he can ftp the logfile back to his own machine. I'll do
1759 a small bit of this and give you a look at the output.
1761 1) Spool stdout to VM reader::
1763 SP PRT TO (another vm guest ) or * for the local vm guest
1765 2) Fill the reader with the trace::
1767 TR IO 7c08-7c09 INST INT CCW PRT RUN
1772 4) Finish the trace::
1776 5) close the reader::
1780 6) list reader contents::
1784 7) copy it to linux4's minidisk::
1786 RECEIVE / LOG TXT A1 ( replace
1789 filel & press F11 to look at it
1790 You should see something like::
1792 00020942' SSCH B2334000 0048813C CC 0 SCH 0000 DEV 7C08
1793 CPA 000FFDF0 PARM 00E2C9C4 KEY 0 FPI C0 LPM 80
1794 CCW 000FFDF0 E4200100 00487FE8 0000 E4240100 ........
1797 00020B0A' I/O DEV 7C08 -> 000197BC' SCH 0000 PARM 00E2C9C4
1798 00021628' TSCH B2354000 >> 00488164 CC 0 SCH 0000 DEV 7C08
1799 CCWA 000FFDF8 DEV STS 0C SCH STS 00 CNT 00EC
1800 KEY 0 FPI C0 CC 0 CTLS 4007
1801 00022238' STSCH B2344000 >> 00488108 CC 0 SCH 0000 DEV 7C08
1803 If you don't like messing up your readed ( because you possibly booted from it )
1804 you can alternatively spool it to another readers guest.
1807 Other common VM device related commands
1808 ---------------------------------------------
1809 These commands are listed only because they have
1810 been of use to me in the past & may be of use to
1811 you too. For more complete info on each of the commands
1812 use type HELP <command> from CMS.
1817 ATT <devno range> <guest>
1819 attach a device to guest * for your own guest
1822 cause VM to issue a fake interrupt.
1824 The VARY command is normally only available to VM administrators::
1826 VARY ON PATH <path> TO <devno range>
1827 VARY OFF PATH <PATH> FROM <devno range>
1829 This is used to switch on or off channel paths to devices.
1831 Q CHPID <channel path ID>
1832 This displays state of devices using this channel path
1834 D SCHIB <subchannel>
1835 This displays the subchannel information SCHIB block for the device.
1836 this I believe is also only available to administrators.
1839 defines a virtual CTC channel to channel connection
1840 2 need to be defined on each guest for the CTC driver to use.
1842 COUPLE devno userid remote devno
1843 Joins a local virtual device to a remote virtual device
1844 ( commonly used for the CTC driver ).
1846 Building a VM ramdisk under CMS which linux can use::
1848 def vfb-<blocksize> <subchannel> <number blocks>
1850 blocksize is commonly 4096 for linux.
1854 format <subchannel> <driver letter e.g. x> (blksize <blocksize>
1856 Sharing a disk between multiple guests::
1858 LINK userid devno1 devno2 mode password
1864 N.B. if compiling for debugging gdb works better without optimisation
1865 ( see Compiling programs for debugging )
1869 gdb <victim program> <optional corefile>
1873 help: gives help on commands
1880 Note gdb's online help is very good use it.
1886 displays registers other than floating point.
1889 displays floating points as well.
1896 disassemble without parameters will disassemble the current function
1897 disassemble $pc $pc+10
1899 Viewing & modifying variables
1900 -----------------------------
1902 displays variable or register
1904 e.g. p/x $sp will display the stack pointer
1907 prints variable or register each time program stops
1911 display/x $pc will display the program counter
1918 shows all current breakpoints
1921 shows stack back trace (if this doesn't work too well, I'll show
1922 you the stacktrace by hand below).
1925 displays local variables.
1928 display current procedure arguments.
1931 will set argc & argv each time the victim program is invoked
1935 set <variable>=value
1944 steps n lines of sourcecode
1950 steps 100 lines of code.
1953 like step except this will not step into subroutines
1956 steps a single machine code instruction.
1963 steps a single machine code instruction but will not step into
1967 will run until exit of the current routine
1970 (re)starts a program
1991 Here's a really useful one for large programs
1994 Set a breakpoint for all functions matching REGEXP
2000 will set a breakpoint with all functions with 390 in their name.
2003 lists all breakpoints
2006 delete breakpoint by number or delete them all
2011 will delete the first breakpoint
2015 will delete them all
2018 This will set a watchpoint ( usually hardware assisted ),
2020 This will watch a variable till it changes
2025 will watch the variable cnt till it changes.
2027 As an aside unfortunately gdb's, architecture independent watchpoint code
2028 is inconsistent & not very good, watchpoints usually work but not always.
2031 Display currently active watchpoints
2033 condition: ( another useful one )
2034 Specify breakpoint number N to break only if COND is true.
2036 Usage is `condition N COND`, where N is an integer and COND is an
2037 expression to be evaluated whenever breakpoint N is reached.
2041 User defined functions/macros
2042 -----------------------------
2043 define: ( Note this is very very useful,simple & powerful )
2045 usage define <name> <list of commands> end
2047 examples which you should consider putting into .gdbinit in your home
2052 disassemble $pc $pc+10
2056 disassemble $pc $pc+10
2060 Other hard to classify stuff
2061 ----------------------------
2063 sends the victim program a signal.
2065 e.g. `signal 3` will send a SIGQUIT.
2068 what gdb does when the victim receives certain signals.
2075 lists current function source
2077 list first 10 lines of current file.
2083 Adds directories to be searched for source if gdb cannot find the source.
2084 (note it is a bit sensitive about slashes)
2086 e.g. To add the root of the filesystem to the searchpath do::
2092 This calls a function in the victim program, this is pretty powerful
2094 (gdb) call printf("hello world")
2098 You might now be thinking that the line above didn't work, something extra had
2100 (gdb) call fflush(stdout)
2102 As an aside the debugger also calls malloc & free under the hood
2103 to make space for the "hello world" string.
2109 1) command completion works just like bash
2110 ( if you are a bad typist like me this really helps )
2112 e.g. hit br <TAB> & cursor up & down :-).
2114 2) if you have a debugging problem that takes a few steps to recreate
2115 put the steps into a file called .gdbinit in your current working directory
2116 if you have defined a few extra useful user defined commands put these in
2117 your home directory & they will be read each time gdb is launched.
2119 A typical .gdbinit file might be.::
2123 break runtime_exception
2127 stack chaining in gdb by hand
2128 -----------------------------
2129 This is done using a the same trick described for VM::
2131 p/x (*($sp+56))&0x7fffffff
2133 get the first backchain.
2136 Replace 56 with 112 & ignore the &0x7fffffff
2137 in the macros below & do nasty casts to longs like the following
2138 as gdb unfortunately deals with printed arguments as ints which
2139 messes up everything.
2141 i.e. here is a 3rd backchain dereference::
2143 p/x *(long *)(***(long ***)$sp+112)
2154 info symbol (*($sp+56))&0x7fffffff
2156 you might see something like::
2158 rl_getc + 36 in section .text
2160 telling you what is located at address 0x528f18
2163 p/x (*(*$sp+56))&0x7fffffff
2171 info symbol (*(*$sp+56))&0x7fffffff
2172 rl_read_key + 180 in section .text
2176 p/x (*(**$sp+56))&0x7fffffff
2180 Disassembling instructions without debug info
2181 ---------------------------------------------
2182 gdb typically complains if there is a lack of debugging
2183 symbols in the disassemble command with
2184 "No function contains specified address." To get around
2187 x/<number lines to disassemble>xi <address>
2196 Remember gdb has history just like bash you don't need to retype the
2197 whole line just use the up & down arrows.
2203 From your linuxbox do::
2217 A core dump is a file generated by the kernel (if allowed) which contains the
2218 registers and all active pages of the program which has crashed.
2220 From this file gdb will allow you to look at the registers, stack trace and
2221 memory of the program as if it just crashed on your system. It is usually
2222 called core and created in the current working directory.
2224 This is very useful in that a customer can mail a core dump to a technical
2225 support department and the technical support department can reconstruct what
2226 happened. Provided they have an identical copy of this program with debugging
2227 symbols compiled in and the source base of this build is available.
2229 In short it is far more useful than something like a crash log could ever hope
2232 Why have I never seen one ?
2233 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2235 Probably because you haven't used the command::
2237 ulimit -c unlimited in bash
2239 to allow core dumps, now do::
2243 to verify that the limit was accepted.
2246 To create this I'm going to do::
2251 to launch gdb (my victim app. ) now be bad & do the following from another
2252 telnet/xterm session to the same machine::
2255 kill -SIGSEGV <gdb's pid>
2257 or alternatively use `killall -SIGSEGV gdb` if you have the killall command.
2259 Now look at the core dump::
2263 Displays the following::
2266 Copyright 1998 Free Software Foundation, Inc.
2267 GDB is free software, covered by the GNU General Public License, and you are
2268 welcome to change it and/or distribute copies of it under certain conditions.
2269 Type "show copying" to see the conditions.
2270 There is absolutely no warranty for GDB. Type "show warranty" for details.
2271 This GDB was configured as "s390-ibm-linux"...
2272 Core was generated by `./gdb'.
2273 Program terminated with signal 11, Segmentation fault.
2274 Reading symbols from /usr/lib/libncurses.so.4...done.
2275 Reading symbols from /lib/libm.so.6...done.
2276 Reading symbols from /lib/libc.so.6...done.
2277 Reading symbols from /lib/ld-linux.so.2...done.
2278 #0 0x40126d1a in read () from /lib/libc.so.6
2279 Setting up the environment for debugging gdb.
2280 Breakpoint 1 at 0x4dc6f8: file utils.c, line 471.
2281 Breakpoint 2 at 0x4d87a4: file top.c, line 2609.
2282 (top-gdb) info stack
2283 #0 0x40126d1a in read () from /lib/libc.so.6
2284 #1 0x528f26 in rl_getc (stream=0x7ffffde8) at input.c:402
2285 #2 0x528ed0 in rl_read_key () at input.c:381
2286 #3 0x5167e6 in readline_internal_char () at readline.c:454
2287 #4 0x5168ee in readline_internal_charloop () at readline.c:507
2288 #5 0x51692c in readline_internal () at readline.c:521
2289 #6 0x5164fe in readline (prompt=0x7ffff810)
2291 #7 0x4d7a8a in command_line_input (prompt=0x564420 "(gdb) ", repeat=1,
2292 annotation_suffix=0x4d6b44 "prompt") at top.c:2091
2293 #8 0x4d6cf0 in command_loop () at top.c:1345
2294 #9 0x4e25bc in main (argc=1, argv=0x7ffffdf4) at main.c:635
2299 This is a program which lists the shared libraries which a library needs,
2300 Note you also get the relocations of the shared library text segments which
2301 help when using objdump --source.
2309 libncurses.so.4 => /usr/lib/libncurses.so.4 (0x40018000)
2310 libm.so.6 => /lib/libm.so.6 (0x4005e000)
2311 libc.so.6 => /lib/libc.so.6 (0x40084000)
2312 /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x40000000)
2315 Debugging shared libraries
2316 ==========================
2317 Most programs use shared libraries, however it can be very painful
2318 when you single step instruction into a function like printf for the
2319 first time & you end up in functions like _dl_runtime_resolve this is
2320 the ld.so doing lazy binding, lazy binding is a concept in ELF where
2321 shared library functions are not loaded into memory unless they are
2322 actually used, great for saving memory but a pain to debug.
2324 To get around this either relink the program -static or exit gdb type
2325 export LD_BIND_NOW=true this will stop lazy binding & restart the gdb'ing
2326 the program in question.
2332 As modules are dynamically loaded into the kernel their address can be
2333 anywhere to get around this use the -m option with insmod to emit a load
2334 map which can be piped into a file if required.
2336 The proc file system
2337 ====================
2339 It is a filesystem created by the kernel with files which are created on demand
2340 by the kernel if read, or can be used to modify kernel parameters,
2341 it is a powerful concept.
2345 cat /proc/sys/net/ipv4/ip_forward
2347 On my machine outputs::
2351 telling me ip_forwarding is not on to switch it on I can do::
2353 echo 1 > /proc/sys/net/ipv4/ip_forward
2357 cat /proc/sys/net/ipv4/ip_forward
2359 On my machine now outputs::
2363 IP forwarding is on.
2365 There is a lot of useful info in here best found by going in and having a look
2366 around, so I'll take you through some entries I consider important.
2368 All the processes running on the machine have their own entry defined by
2371 So lets have a look at the init process::
2384 This contains numerical entries of all the open files,
2385 some of these you can cat e.g. stdout (2)::
2389 on my machine emits::
2391 00400000-00478000 r-xp 00000000 5f:00 4103 /bin/bash
2392 00478000-0047e000 rw-p 00077000 5f:00 4103 /bin/bash
2393 0047e000-00492000 rwxp 00000000 00:00 0
2394 40000000-40015000 r-xp 00000000 5f:00 14382 /lib/ld-2.1.2.so
2395 40015000-40016000 rw-p 00014000 5f:00 14382 /lib/ld-2.1.2.so
2396 40016000-40017000 rwxp 00000000 00:00 0
2397 40017000-40018000 rw-p 00000000 00:00 0
2398 40018000-4001b000 r-xp 00000000 5f:00 14435 /lib/libtermcap.so.2.0.8
2399 4001b000-4001c000 rw-p 00002000 5f:00 14435 /lib/libtermcap.so.2.0.8
2400 4001c000-4010d000 r-xp 00000000 5f:00 14387 /lib/libc-2.1.2.so
2401 4010d000-40111000 rw-p 000f0000 5f:00 14387 /lib/libc-2.1.2.so
2402 40111000-40114000 rw-p 00000000 00:00 0
2403 40114000-4011e000 r-xp 00000000 5f:00 14408 /lib/libnss_files-2.1.2.so
2404 4011e000-4011f000 rw-p 00009000 5f:00 14408 /lib/libnss_files-2.1.2.so
2405 7fffd000-80000000 rwxp ffffe000 00:00 0
2408 Showing us the shared libraries init uses where they are in memory
2409 & memory access permissions for each virtual memory area.
2411 /proc/1/cwd is a softlink to the current working directory.
2413 /proc/1/root is the root of the filesystem for this process.
2415 /proc/1/mem is the current running processes memory which you
2416 can read & write to like a file.
2418 strace uses this sometimes as it is a bit faster than the
2419 rather inefficient ptrace interface for peeking at DATA.
2439 SigPnd: 0000000000000000
2440 SigBlk: 0000000000000000
2441 SigIgn: 7fffffffd7f0d8fc
2442 SigCgt: 00000000280b2603
2443 CapInh: 00000000fffffeff
2444 CapPrm: 00000000ffffffff
2445 CapEff: 00000000fffffeff
2447 User PSW: 070de000 80414146
2448 task: 004b6000 tss: 004b62d8 ksp: 004b7ca8 pt_regs: 004b7f68
2450 00000400 00000000 0000000b 7ffffa90
2451 00000000 00000000 00000000 0045d9f4
2452 0045cafc 7ffffa90 7fffff18 0045cb08
2453 00010400 804039e8 80403af8 7ffff8b0
2455 00000000 00000000 00000000 00000000
2456 00000001 00000000 00000000 00000000
2457 00000000 00000000 00000000 00000000
2458 00000000 00000000 00000000 00000000
2459 Kernel BackChain CallChain BackChain CallChain
2460 004b7ca8 8002bd0c 004b7d18 8002b92c
2461 004b7db8 8005cd50 004b7e38 8005d12a
2464 Showing among other things memory usage & status of some signals &
2465 the processes'es registers from the kernel task_structure
2466 as well as a backchain which may be useful if a process crashes
2467 in the kernel for some unknown reason.
2469 Some driver debugging techniques
2470 ================================
2473 Some of our drivers now support a "debug feature" in
2474 /proc/s390dbf see s390dbf.txt in the linux/Documentation directory
2478 to switch on the lcs "debug feature"::
2480 echo 5 > /proc/s390dbf/lcs/level
2482 & then after the error occurred::
2484 cat /proc/s390dbf/lcs/sprintf >/logfile
2486 the logfile now contains some information which may help
2487 tech support resolve a problem in the field.
2491 high level debugging network drivers
2492 ------------------------------------
2493 ifconfig is a quite useful command
2494 it gives the current state of network drivers.
2496 If you suspect your network device driver is dead
2497 one way to check is type::
2499 ifconfig <network device>
2503 You should see something like::
2506 tr0 Link encap:16/4 Mbps Token Ring (New) HWaddr 00:04:AC:20:8E:48
2507 inet addr:9.164.185.132 Bcast:9.164.191.255 Mask:255.255.224.0
2508 UP BROADCAST RUNNING MULTICAST MTU:2000 Metric:1
2509 RX packets:246134 errors:0 dropped:0 overruns:0 frame:0
2510 TX packets:5 errors:0 dropped:0 overruns:0 carrier:0
2511 collisions:0 txqueuelen:100
2513 if the device doesn't say up
2516 /etc/rc.d/init.d/network start
2518 ( this starts the network stack & hopefully calls ifconfig tr0 up ).
2519 ifconfig looks at the output of /proc/net/dev and presents it in a more
2522 Now ping the device from a machine in the same subnet.
2524 if the RX packets count & TX packets counts don't increment you probably
2531 Do you see any hardware addresses in the cache if not you may have problems.
2534 ping -c 5 <broadcast_addr>
2536 i.e. the Bcast field above in the output of
2537 ifconfig. Do you see any replies from machines other than the local machine
2538 if not you may have problems. also if the TX packets count in ifconfig
2539 hasn't incremented either you have serious problems in your driver
2540 (e.g. the txbusy field of the network device being stuck on )
2541 or you may have multiple network devices connected.
2546 There is a new device layer for channel devices, some
2547 drivers e.g. lcs are registered with this layer.
2549 If the device uses the channel device layer you'll be
2550 able to find what interrupts it uses & the current state
2553 See the manpage chandev.8 &type cat /proc/chandev for more info.
2558 This is now supported by linux for s/390 & z/Architecture.
2560 To enable it do compile the kernel with::
2562 Kernel Hacking -> Magic SysRq Key Enabled
2566 echo "1" > /proc/sys/kernel/sysrq
2570 echo "8" >/proc/sys/kernel/printk
2572 To make printk output go to console.
2574 On 390 all commands are prefixed with::
2580 ^-t will show tasks.
2581 ^-? or some unknown command will display help.
2583 The sysrq key reading is very picky ( I have to type the keys in an
2584 xterm session & paste them into the x3270 console )
2585 & it may be wise to predefine the keys as described in the VM hints above
2587 This is particularly useful for syncing disks unmounting & rebooting
2588 if the machine gets partially hung.
2590 Read Documentation/admin-guide/sysrq.rst for more info
2594 - Enterprise Systems Architecture Reference Summary
2595 - Enterprise Systems Architecture Principles of Operation
2596 - Hartmut Penners s390 stack frame sheet.
2597 - IBM Mainframe Channel Attachment a technology brief from a CISCO webpage
2598 - Various bits of man & info pages of Linux.
2599 - Linux & GDB source.
2600 - Various info & man pages.
2601 - CMS Help on tracing commands.
2602 - Linux for s/390 Elf Application Binary Interface
2603 - Linux for z/Series Elf Application Binary Interface ( Both Highly Recommended )
2604 - z/Architecture Principles of Operation SA22-7832-00
2605 - Enterprise Systems Architecture/390 Reference Summary SA22-7209-01 & the
2606 - Enterprise Systems Architecture/390 Principles of Operation SA22-7201-05
2610 Special thanks to Neale Ferguson who maintains a much
2611 prettier HTML version of this page at
2612 http://linuxvm.org/penguinvm/
2613 Bob Grainger Stefan Bader & others for reporting bugs
git.samba.org / sfrench / cifs-2.6.git / blob