Dynamic Tracing (DTrace)

Dynamic Tracing in the Solaris Operating System! If you have ever wanted to understand the behavior of your system, DTrace is the tool for you.

DTrace is a comprehensive dynamic tracing facility that is built into Solaris that can be used by administrators and developers on live production systems to examine the behavior of both user programs and of the operating system itself.

DTrace enables you to explore your system to understand how it works, track down performance problems across many layers of software, or locate the cause of aberrant behavior. As you’ll see, DTrace lets you create your own custom programs to dynamically instrument the system and provide immediate, concise answers to arbitrary questions you can formulate using the DTrace D programming language.

The first section of this chapter provides a quick introduction to DTrace and shows you how to write your very first D program. The rest of the chapter introduces the complete set of rules for programming in D as well as tips and techniques for performing in-depth analysis of your system.

You can share your DTrace experiences and scripts with the rest of the DTrace community on the web at http://www.opensolaris.org/os/community/dtrace/ and http://www.sun.com/bigadmin/content/dtrace/. All of the example scripts presented in this guide can be found on your Solaris system in the directory /usr/demo/dtrace.

Click here to see more information on DTrace.

stty – Sets options for your terminal

Syntax

stty [-a] [-g] [modes]

-a
Write to standard output all of the option set tings for the terminal.

-g
Report current settings in a form that can be used as an argument to another stty command. Emits termios-type output if the underlying driver sup ports it; otherwise, it emits termio-type output.

modes
The following mode operands are supported: parenb(-parenb) Enable (disable) parity generation and detection.

parext(-parext)
Enable (disable) extended parity generation and detection for mark and space parity.

parodd(-parodd)
Select odd (even) parity, or mark (space) parity if parext is enabled.

cs5 cs6 cs7 cs8
Select character size.

0
Hang up line immediately.

hupcl (-hupcl)
Hang up (do not hang up) connection on last close.

hup (-hup)
Same as hupcl(-hupcl).

cstopb (-cstopb)
Use two (one) stop bits per character.

cread (-cread)
Enable (disable) the receiver.

crtscts (-crtscts)
Enable output hardware flow control. Raise the RTS (Request to Send) modem control line. Suspends output until the CTS (Clear to Send) line is raised.

crtsxoff (-crtsxoff)
Enable input hardware flow control. Raise the RTS (Request to Send) modem control line to receive data. Suspends input when RTS is low.

clocal (-clocal)
Assume a line without (with) modem control.

loblk (-loblk)
Block (do not block) output from a non-current layer.

defeucw
Set the widths of multibyte Extended Unix Code (EUC) characters in struct eucioc to default values for the current locale specified by LC_CTYPE; width is expressed in
terms of bytes per character, and screen or display columns per character.

110 300 600 1200 1800 2400 4800 9600 19200 38400 357600 76800 115200 153600 230400 307200 460800
Set terminal baud rate to the number given, if possible. (All speeds are not supported by all hardware interfaces.)

ispeed 0 110 300 600 1200 1800 2400 4800 9600 19200 38400 57600 76800 115200″ 10 153600 230400 307200 460800
Set terminal input baud rate to the number given, if possible. (Not all hardware supports split baud rates.) If the input baud rate is set to  0, the input baud rate will be specified by the value of the output baud rate.

ospeed 0 110 300 600 1200 1800 2400 4800 9600 19200 38400 57600 76800 115200″ 10 153600 230400 307200 460800
Set terminal output baud rate to the number given, if possible. (Not all hardware supports split baud rates.) If the output baud rate is set to 0, the line will be hung up immediately.

ignbrk (-ignbrk)
Ignore (do not ignore) break on input.

brkint (-brkint)
Signal (do not signal) INTR on break.

ignpar (-ignpar)
Ignore (do not ignore) parity errors.

parmrk (-parmrk)
Mark (do not mark) parity errors.

inpck (-inpck)
Enable (disable) input parity checking.

istrip (-istrip)
Strip (do not strip) input characters to seven bits.

inlcr (-inlcr)
Map (do not map) NL to CR on input.

igncr (-igncr)
Ignore (do not ignore) CR on input.

icrnl (-icrnl)
Map (do not map) CR to NL on input.

iuclc (-iuclc)
Map (do not map) upper-case alphabetics to lower case on input.

ixon (-ixon)
Enable (disable) START/STOP output control. Out put is stopped by sending STOP control character and started by sending the START control character.

ixany (-ixany)
Allow any character (only DC1) to restart output.

ixoff (-ixoff)
Request that the system send (not send) START/STOP characters when the input queue is nearly empty/full.

imaxbel (-imaxbel)
Echo (do not echo) BEL when the input line is too long.

opost (-opost)
Post-process output (do not post-process output; ignore all other output modes).

olcuc (-olcuc)
Map (do not map) lower-case alphabetics to upper case on output.

onlcr (-onlcr)
Map (do not map) NL to CR-NL on output.

ocrnl (-ocrnl)
Map (do not map) CR to NL on output.

onocr (-onocr)
Do not (do) output CRs at column zero.

onlret (-onlret)
On the terminal NL performs (does not perform) the CR function.

ofill (-ofill)
Use fill characters (use timing) for delays.

ofdel (-ofdel)
Fill characters are DELs (NULs).

cr0 cr1 cr2 cr3
Select style of delay for carriage returns.

nl0 nl1
Select style of delay for line-feeds.

tab0 tab1 tab2 tab3
Select style of delay for horizontal tabs.

bs0 bs1
Select style of delay for backspaces.

ff0 ff1
Select style of delay for form-feeds.

vt0 vt1
Select style of delay for vertical tabs.

isig(-isig)
Enable (disable) the checking of characters against the special control characters INTR, QUIT, SWTCH, and SUSP.

icanon (-icanon)
Enable (disable) canonical input (ERASE and KILL processing). Does not set MIN or TIME.

xcase (-xcase)
Canonical (unprocessed) upper/lower-case presentation.

echo (-echo)
Echo back (do not echo back) every character typed.

echoe (-echoe)
Echo (do not echo) ERASE character as a backspace-space-backspace string. Note: This mode will erase the ERASEed character on many CRT terminals; however, it does not keep track of column position and, as a result, it may be confusing for escaped characters, tabs, and backspaces.

echok(-echok)
Echo (do not echo) NL after KILL character.

lfkc (-lfkc)
The same as echok(-echok); obsolete.

echonl (-echonl)
Echo (do not echo) NL.

noflsh (-noflsh)
Disable (enable) flush after INTR, QUIT, or SUSP.

stwrap (-stwrap)
Disable (enable) truncation of lines longer than 79 characters on a synchronous line.

tostop (-tostop)
Send (do not send) SIGTTOU when background processes write to the terminal.

echoctl (-echoctl)
Echo (do not echo) control characters as ^char, delete as ^?.

echoprt (-echoprt)
Echo (do not echo) erase character as character is “erased”.

echoke (-echoke)
BS-SP-BS erase (do not BS-SP-BS erase) entire line on line kill.

flusho (-flusho)
Output is (is not) being flushed.

pendin (-pendin)
Retype (do not retype) pending input at next read or input character.

iexten (-iexten)
Enable (disable) special control characters not currently controlled by icanon, isig, ixon, or ixoff: VEOLZ, VSWTCH, VREPRINT, VDISCARD, VDSUSP, VWERASE, and VLNEXT.

stflush (-stflush)
Enable (disable) flush on a synchronous line after every write.

stappl (-stappl)
Use application mode (use line mode) on a synchro nous line.

rtsxoff (-rtsxoff)
Enable (disable) RTS hardware flow control on input.

ctsxon (-ctsxon)
Enable (disable) CTS hardware flow control on out put.

dtrxoff (-dtrxoff)
Enable (disable) DTR hardware flow control on input.

cdxon (-cdxon)
Enable (disable) CD hardware flow control on out put.

isxoff (-isxoff)
Enable (disable) isochronous hardware flow control on input.

xcibrg
Get transmit clock from internal baud rate generator.

xctset
Get the transmit clock from transmitter signal element timing (DCE source) lead, CCITT V.24 cir cuit 114, EIA-232-D pin 15.

xcrset
Get transmit clock from receiver signal element timing (DCE source) lead, CCITT V.24 circuit 115, EIA-232-D pin 17.

rcibrg
Get receive clock from internal baud rate generator.

rctset
Get receive clock from transmitter signal element timing (DCE source) lead, CCITT V.24 circuit 114, EIA-232-D pin 15.

rcrset
Get receive clock from receiver signal element timing (DCE source) lead, CCITT V.24 circuit 115, EIA-232-D pin 17.

tsetcoff
Transmitter signal element timing clock not pro vided.

tsetcrbrg
Output receive baud rate generator on transmitter signal element timing (DTE source) lead, CCITT V.24 circuit 113, EIA-232-D pin 24.

tsetctset
Output transmitter signal element timing (DCE source) on transmitter signal element timing (DTE source) lead, CCITT V.24 circuit 113, EIA-232-D pin 24.

tsetcrset
Output receiver signal element timing (DCE source) on transmitter signal element timing (DTE source) lead, CCITT V.24 circuit 113, EIA-232-D pin 24.

rsetcoff
Receiver signal element timing clock not provided.

rsetcrbrg
Output receive baud rate generator on receiver signal element timing (DTE source) lead, CCITT V.24 circuit 128, no EIA-232-D pin.

rsetctbrg
Output transmit baud rate generator on receiver signal element timing (DTE source) lead, CCITT V.24 circuit 128, no EIA-232-D pin.

rsetctset
Output transmitter signal element timing (DCE source) on receiver signal element timing (DTE source) lead, CCITT V.24 circuit 128, no EIA-232-D pin.

rsetcrset
Output receiver signal element timing (DCE source) on receiver signal element timing (DTE source) lead, CCITT V.24 circuit 128, no EIA-232-D pin.

 
Examples

stty
typing stty alone on a dialup connection would list the information about the connection. Below is an example of what may be displayed.

speed 38400 baud; -parity
rows = 49; columns = 132; ypixels = 0; xpixels = 0;
erase = ^h; swtch = <undef>;
brkint -inpck -istrip icrnl -ixany imaxbel onlcr tab3
echo echoe echok echoctl echoke iexten 

stty erase \^h
sets the erase key to backspace. 

stty sane
resets the terminal’s settings  to values that the shell believes they should be. If encountering issues with terminal or typing in remote connection this command should be used.

pwd – displays the name of the current working directory

Syntax

pwd

Examples

pwd
Typing pwd at the prompt would give you something similar to:

$ pwd
/home/oravis

Users who are familiar with MS-DOS or the Windows command prompt may type cd alone to print the working directory. However, typing cd alone in Linux / Unix will return you to the home directory.

sh – Runs or processes jobs through the Bourne shell

Syntax

sh [-a] [-c] [-C] [-e] [-E] [-f] [-h] [-i] [-I][-k] [-m] [-n] [-p] [-r] [-s] [-t] [-T] [-u] [-v] [-x] [ argument ]

-a
Export all variables assigned to.

-c
Pass the string argument to the shell to be interpreted as input. Keep in mind that this option only accepts a single string as its argument, hence multi-word strings must be quoted.

-C
Don’t overwrite existing files with “>.”

-e
If not interactive, exit immediately if any untested command fails. The exit status of a command is considered to be explic- itly tested if the command is used to control an if, elif, while, or until; or if the command is the left hand operand of an “&&” or “||” operator.

-E
Enable the built-in emacs command line editor (disables -V if it has been set).

-f
Disable pathname expansion.

-h
Makes all commands use tracked aliases.

-i
Force the shell to behave interactively.

-I
Ignore EOF’s from input when interactive.

-k
tells the shell to use Korn-compatible behavior in any case where the POSIX.2 behavior is different from the behavior specified by Korn. In particular, this affects the trap command.

-m
Turn on job control (set automatically when interactive).

-n
If not interactive, read commands but do not execute them. This is useful for checking the syntax of shell scripts.

-p
Turn on privileged mode. This mode is enabled on startup if either the effective user or group id is not equal to the real user or group id. Turning this mode off sets the effective user and group ids to the real user and group ids. Also on interactive shells and when enabled, this mode sources /etc/suid_profile (in- stead of
~/.profile) after /etc/profile and ignores the contents of the ENV variable.

-r
Invokes a restricted shell. In a restricted shell, you cannot do any of the following: use the cd command; change the values of the variables env, path or shell; use > or >> to redirect output; specify command names containing /. These restrictions do not apply during execution of profile files.

-s
Read commands from standard input (set automatically if no file arguments are present). This option has no effect when set after the shell has already started running (i.e. with set).

-t
Exits after reading and executing one command.

-T
When waiting for a child, execute traps immediately. If this option is not set, traps are executed after the child exits, as specified in IEEE Std1003.2 (“POSIX”) This nonstandard option is useful to put guarding shells around childs that block signals. The surrounding shell may kill the child or it may just re- turn control to the tty and leave the child alone.

-u
Write a message to standard error when attempting to expand a variable that is not set, and if the shell is not interactive, exit immediately.

-v
The shell writes its input to standard error as it is read. Useful for debugging.

-V
Enable the built-in vi command line editor (disables -E if it has been set).

-x
Write each command to standard error (preceded by a ‘+ ‘) before it is executed. Useful for debugging.

Examples

sh – Executes the Bourne shell, likely taking you to a $ prompt.

at – Schedules a command to be ran at a particular time.

Syntax

at
executes commands at a specified time.

atq
lists the user’s pending jobs, unless the user is the superuser; in that case, everybody’s jobs are listed. The format of the output lines (one for each job) is: Job number, date, hour, job class.

atrm
deletes jobs, identified by their job number.

batch
executes commands when system load levels permit; in other words, when the load average drops below 1.5, or the value specified in the invocation of atrun.

at [-c | -k | -s] [-f filename] [-q queuename] [-m] -t time [date] [-l] [-r]

-c
C shell. csh(1) is used to execute the at-job.

-k
Korn shell. ksh(1) is used to execute the at-job.

-s
Bourne shell. sh(1) is used to execute the at-job.

-f
filename Specifies the file that contains the command to run.

-m
Sends mail once the command has been run.

-t time
Specifies at what time you want the command to be ran. Format hh:mm. am / pm indication can also follow the time otherwise a 24-hour clock is used. A timezone
name of GMT, UCT or ZULU (case insensitive) can follow to specify that the time is in Coordinated Universal Time. Other timezones can be specified using the TZ environment variable. The below quick times can also be entered:

midnight – Indicates the time 12:00 am (00:00).
noon – Indicates the time 12:00 pm.
now – Indicates the current day and time. Invoking at – now will submit submit an at-job for potentially immediate execution.

date
Specifies the date you wish it to be ran on. Format month, date, year. The following quick days can also be entered:

today – Indicates the current day.
tomorrow – Indicates  the day following the current day.

-l
Lists the commands that have been set to run.

-r
Cancels the command that you have set in the past.

Examples

at -m 01:35 < atjob
Run the commands listed in the ‘atjob’ file at 1:35AM, in addition all output that is generated from job mail to the user running the task. When this command has been successfully enter you should receive a prompt similar to the below example.

commands will be executed using /bin/csh
job 1072250520.a at Wed Dec 24 00:22:00 2003

at -l
This command will list each of the scheduled jobs as seen below.

1072250520.a Wed Dec 24 00:22:00 2003

at -r 1072250520.a
Deletes the job just created. 

  or

atrm 23
Deletes job 23.

If you wish to create a job that is repeated you could modify the file that executes the commands with another command that recreates the job or better yet use the crontab command.

Note: Performing just the at command at the prompt will give you an error “Garbled Time”, this is a standard error message if no switch or time setting is given.

crontab – List of files that you want to run on a regular schedule

Syntax

crontab [-e] [-l] [-r] [filename]

-e
edit a copy of the current user’s crontab file, or creates an empty file to edit if crontab does not exist. When editing is complete, the file is installed as the user’s crontab file. If a user- name is given, the specified user’s crontab file is edited, rather than the current user’s crontab file; this may only be done by a super-user. The environment variable EDITOR determines which editor is invoked with the -e option. The default editor is ed. Note that all crontab jobs should be submitted using crontab ; you should not add jobs by just editing the crontab file because cron will not be aware of changes made this way.

-l
list the crontab file for the invoking user. Only a super-user can specify a username following the -r or -l options to remove or list the crontab file of the specified user.

-r
remove a user’s crontab from the crontab

filename
The filename that contains the commands to run.
Lines that can be in the crontab file.

minute (0-59),
hour (0-23),
day of the month (1-31),
month of the year (1-12),
day of the week (0-6 with 0=Sunday).

jobs – Lists the jobs that you are running in the background and in the foreground

Syntax

jobs [-p | -l] [-n] [-p] [-x] [job id]

-p | -l Report the process group ID and working directory of the jobs.

-n Display only jobs that have stopped or exited since last notified.

-p Displays only the process IDs for the process group leaders of the selected jobs.

-x Replace any job_id found in command or arguments with the corresponding process group ID, and then execute command passing it arguments.
job id The job id.

Examples:

jobs

Would display results similar to the below if jobs were running in the background.

[1] + Stopped (user)      man jobs

As you can see in the above job example the id is 1 it has been stopped by the user and the process in this case is man jobs (looking at the manual for jobs).

jobs -l

The above command would not just list the jobs running but also this group ID and the working directory of the jobs. Below is an example of what this would display.

[3] 16882 Running ./chsearchproc (wd: ~/public_html/cgi-bin/chsearch)

kill – Cancels a job

Syntax

kill [-s] [-l] %pid

-s Specify the signal to send, using one of the symbolic names defined in the <signal.h> description. Values of signal will be recognized in a case independent fashion, without the SIG prefix. In addition, the symbolic name 0 will be recognized, representing the signal value zero. The corresponding signal will be sent instead of SIGTERM.

-l Write all values of signal sup ported by the implementation, if no operand is given. If an exit_status operand is given and it is a value of the ? shell special parameter and wait corresponding to a process that was ter minated by a signal, the signal corresponding to the signal that terminated the process will be written. If an exit_status operand is given and it is the unsigned decimal integer value of a signal number, the signal corresponding to that signal will be written. Otherwise, the results are unspecified.

pid
One of the following:
1. A decimal integer specifying a process or process group to be signaled. The process or processes selected by positive, negative and zero values of the pid operand will be as
described for the kill function. If process number 0 is specified, all processes in the process group are signaled. If the first pid operand is negative, it should be preceded by — to keep it from being interpreted as an option.

2. A job control job ID that identifies a background process group to be signaled. The job control job ID notation is applicable only for invocations of kill in the current shell execution environment.

Note the job control job ID type of pid is available only on systems supporting the job control option.

ps – Reports the process status

Syntax

ps [-a] [-A] [-c] [-d] [-e] [-f] [-j] [-l] [-L] [-P] [-y] [ -g grplist ] [ -n namelist ] [-o format ] [ -p proclist ] [ -s sidlist ] [ -t term] [ -u uidlist ] [ -U uidlist ] [ -G gidlist ]

-a List information about all processes most frequently requested: all those except process group leaders and processes not associated with a terminal.

-A List information for all processes. Identical to -e, below.

-c Print information in a format that reflects scheduler properties as described in priocntl. The -c option affects the output of the -f and -l options, as described below.

-d List information about all processes except session leaders.

-e List information about every process now running.

-f Generate a full listing.

-j Print session ID and process group ID.

-l Generate a long listing.

-L Print information about each light weight process (lwp) in each selected process.

-P Print the number of the processor to which the process or lwp is bound, if any, under an additional column header, PSR.

-y Under a long listing (-l), omit the obsolete F and ADDR columns and include an RSS column to report the resident set size of the process. Under the -y option, both RSS and SZ will be reported in units of kilobytes instead of pages.

-g grplist List only process data whose group leader’s ID number(s) appears in grplist. (A group leader is a process whose process ID number is identical to its process group ID number.)

-n namelist Specify the name of an alternative system namelist file in place of the default. This option is accepted for compatibility, but is ignored.

-o format Print information according to the format specification given in format. This is fully described in DISPLAY FORMATS. Multiple -o options can be specified; the format specification will be interpreted as the space-character-separated concatenation of all the format option-arguments.

-p proclist List only process data whose process ID numbers are given in proclist.

-s sidlist List information on all session leaders whose IDs appear in sidlist.

-t term List only process data associated with term. Terminal identifiers are specified as a device file name, and an identifier. For example, term/a, or pts/0.

-u uidlist List only process data whose effective user ID number or login name is given in uidlist. In the listing, the numerical user ID will be printed unless you give the -f option, which prints the login name.

-U uidlist List information for processes whose real user ID numbers or login names are given in uidlist. The uidlist must be a single argument in the form of a blank- or comma-separated list.

-G gidlist List information for processes whose real group ID numbers are given in gidlist. The gidlist must be a single argument in the form of a blank- or comma-separated list.

top – Display UNIX tasks.

top -hv | -bcisS -d delay -n iterations -p pid [, pid …]

The traditional switches ‘-‘ and whitespace are optional.

-b Batch mode operation
Starts top in ‘Batch mode’, which could be useful for sending output from top to other programs or to a file. In this mode, top will not accept input and runs until the iterations limit you’ve set with the ‘-n’ command-line option or until killed.

-c Command line/Program name toggle
Starts top with the last remembered ‘c’ state reversed. Thus, if top was displaying command lines, now that field will show program names, and visa versa. See the ‘c’ interactive command for additional information.

-d Delay time interval as: -d ss.tt (seconds.tenths)
Specifies the delay between screen updates, and overrides the corresponding value in one’s personal configuration file or the startup default. Later this can be changed with the ‘d’ or ‘s’ interactive commands.

Fractional seconds are honored, but a negative number is not allowed. In all cases, however, such changes are prohibited if top is running in ‘Secure mode’, except for root (unless the ‘s’ command-line option was used). For additional information on ‘Secure mode’ see topic 5a. SYSTEM Configuration File.
 
-h Help
Show library version and the usage prompt, then quit.

-i Idle Processes toggle
Starts top with the last remembered ‘i’ state reversed. When this toggle is Off, tasks that are idled or zombied will not be displayed.

-n Number of iterations limit as: -n number
Specifies the maximum number of iterations, or frames, top should produce before ending.

-u Monitor by user as: -u somebody
Monitor only processes with an effective UID or user name matching that given.

-U Monitor by user as: -U somebody
Monitor only processes with a UID or user name matching that given. This matches real, effective, saved, and filesystem UIDs.

-p Monitor PIDs as: -pN1 -pN2 … or -pN1, N2 [,…]
Monitor only processes with specified process IDs. This option can be given up to 20 times, or you can provide a comma delimited list with up to 20 pids. Co-mingling both approaches is permitted.
This is a command-line option only. And should you wish to return to normal operation, it is not necessary to quit
and and restart top — just issue the ‘=’ interactive command.
 
-s Secure mode operation
Starts top with secure mode forced, even for root. This mode is far better controlled through the system configuration file (see topic 5. FILES).

-S Cumulative time mode toggle
Starts top with the last remembered ‘S’ state reversed. When ‘Cumulative mode’ is On, each process is listed with the cpu time that it and its dead children have used. See the ‘S’ interactive command for additional information regarding this mode.

-v Version
Show library version and the usage prompt, then quit.