AD Controller

Manages parallel workers in AD Administration and AutoPatch.

AD Controller is a general maintenance utility you use to determine the status of AutoUpgrade, AD Administration, or AutoPatch workers and to control worker operation. AD Controller is run in its own window, not in the same window as AutoUpgrade, AD Administration, or AutoPatch.

The command to start AD Controller is ‘adctrl’.

AD Controller prompts you for standard items such as the AOL username and password, and the log file name. AD Controller writes its log file to the current working directory.

Reviewing Worker Status

Status	Description
ASSIGNED	The Manager assigned a job to the worker and the worker has not started.
COMPLETED	The worker completed the job and the manager has not yet assigned it a new job
FAILED	The worker encountered a problem
FIXED, RESTART	You fixed the problem and the worker should retry whatever failed
RESTARTED	The worker is retrying a job or has successfully restarted a job
RUNNING	The worker is running a job
WAIT	The worker is idle

Worker Status Flow

Resolving a Failed Worker
The first time a job fails; the manager automatically defers it to the end of the current phase and assigns a new job to the worker. If the deferred job fails the second time it is run, the manager defers it again only if the total runtime of the job is less than ten minutes. If the deferred job fails a third time (or if the job’s total runtime is not less than ten minutes the second time it is run) the job stays at failed status and the worker waits. At this point, you must address the cause of the failure, and then restart the job using AD Controller. If you enabled the email feature when starting the AD utility that started the workers, you automatically receive an email when a worker fails.

Determining Why a Worker Failed
Perform the following steps to investigate the problem that caused the failure and restart a failed worker:

1. Review worker status and confirm the failed status of the worker
   
2. Review the worker log file adworkXXX.log under $APPL_TOP/admin/<SID>/log too determine the source of the error
   
3. Resolve the error.
   

Restarting a Failed Worker
after you have resolved the error

1. Tell the worker to restart a failed job
   
2. When prompted enter the number of the worker that failed
   
3. Review worker status again
   

Restarting a Failed Patch Process
There may be cases where a worker fails and cannot be restarted, for example, a worker process may dump core. In this case the worker is listed as running, but there is actually no worker process present. If this occurs, the only alternative may be to stop the patch and restart. In such as case, you must first shut down all workers manually. After you shut down the workers, the manager must be told that the non-running worker has failed its job and acknowledges quit.    
The progression of AD Controller commands is:

• Option 3: Tell worker to shut down/quit (all workers)
• Option 4: Tell manager that a worker failed its job
• Option 5: Tell manager that a worker acknowledges quit

Once AutoPatch shuts down, restart the patch process.

Terminating a Hanging Worker Process
When running the AD utilities, there may be situations when a worker process appears to hang, or stop processing. If this occurs, and you are satisfied that the process is not simply carrying out a long-running job, you may need to terminate the process manually. Once you do, you must also manually restart that process. In such a case:

1. Determine what the worker process is doing. Use the AD Controller worker status screen to determine the file being processed and check the worker log file to see what it is doing:

– Verify whether the process is consuming CPU.
– Review the file to see what actions are being taken.
– Check for correct indexes on the tables (if the problem appears to be performance related).
– Check for an entry for this process in the V$SESSION table. This may provide clues to what the process is doing in the database.

2. Get the worker’s process ID.

If the job is identified as hanging, determine the worker’s process ID.
– UNIX: $ ps -a | grep adworker
– Windows: Run Task Manager (for example, using Ctrl-Alt-Delete) to view processes.

3. Determine what processes the worker has started, if any. If there are child processes, obtain their process IDs. Examples of child processes include SQL*Plus and FNDLOAD.

4. Stop the hanging process, using the command that is appropriate for your operating system.

5. Make necessary changes. Fix the issue that caused the worker to hang. Contact Oracle Support Services if you do not understand how to proceed.

6. Restart the job or the worker.

Restarting a Terminated Worker
To restart a terminated worker process, complete these steps:
1. Start AD Controller.
2. Review worker status.
3. Take the appropriate action for each worker status:
If the worker shows Failed, restart the failed job. When prompted, enter the number of the worker that failed. If the worker shows Running or Restarted status, but the process is not really running, select the following options:

• Option 4: Tell the manager that a worker has failed its job. When prompted, enter the number of the hanging worker.
• Option 6: Restart a worker on the current machine. When prompted, enter the number of the worker that failed.

Do not tell the manager to start a worker that has shut down if the worker process is running. Doing so will create duplicate worker processes with the same worker ID, which will give unpredictable results.

Restarting a Terminated Child Process

• Some worker processes spawn other processes called child processes
  
• If you terminate child process that is hanging, the worker that spawned the process shows failed as the status
  
• After you fix the problem choose to restart the failed job
  
• Once the worker is restarted the associated child processes are started as well.
  

Restarting an AD Utility after a Machine Crash
If your system crashes while running an AD utility you should:

1. Start AD controller
   
2. Select the following options:
   
   1. Option 4: Tell manager that a worker has failed its job (specify all workers)
      
   2. Option 2:Tell worker to restart a failed job(specify all workers)
      
3. Restart the AD utility that was running when the machine crashed.
   

Because the AD utilities cannot automatically detect a machine crash, you must manually notify the manager that all jobs have failed and manually restart the workers using AD Controller. If you restart the utility without doing this, the utility status and the system status will not be synchronized.

Shutting Down Managers
There may be situations when you need to shut down an AD utility (the manager) while it is running. For example, you may need to shut down your database while you are running AutoPatch, AutoUpgrade, or AD Administration.
You should perform this shutdown in an orderly fashion so that it does not affect your data.
The best way to do this is to shut down the workers manually, which also causes the AD utility to quit in an orderly manner.

Running AD Controller Interactively
Follow these steps to access AD Controller.
1. Log in as applmgr and set the environment as described in Setting the Environment, page 1-24 in this chapter.
2. Start AD Controller with the adctrl command. This will prompt you to:

• Confirm the value of APPL_TOP.
• Specify an AD Controller log file (the default is adctrl.log). The AD Controller log file is written in the current working directory.
• Supply the Oracle Application Object Library user name and password.

3. Choose an option from the main menu. Once you respond to the prompts, the main menu appears.

AD Controller Menu

Type a number to select an option. Press [Return] at any time to return to the AD Controller main menu.

Running AD Controller Non-Interactively
you can run AD Controller without user intervention by creating a defaults file, which captures information you supply at the interactive prompts in a file that you can later use to run AD Controller without user intervention. Creating a defaults file and running AD Controller non-interactively works in much the same way as it does for AD Administration.

Like AD Administration, the same defaults file can be used to run different AD Controller commands: a single file can contain all your choices for the different menu options. In order to choose which task the defaults file will run, you add menu_ option= <menu choice> to the utility start command. This overrides any menu-specific key stroke information stored in the defaults file initially, and allows you to use the defaults file for any of the AD Controller menu items. It also ensures that the menu option you intended for the defaults file is always valid, even it the menu items are renumbered or relocated in subsequent releases.

The available options are listed in the following table.

AD Controller Menu Options

Menu Option Effect

ACKNOWLEDGE_QUIT Tell manager that a worker acknowledges quit
INFORM_FAILURE Tell manager that a worker failed its job
RESTART_JOB Tell worker to restart a failed job
SHOW_STATUS Show worker status
SHUTDOWN_WORKER Tell worker to quit
START_WORKER Restart a worker on the current machine

The following is an example of running AD Controller non-interactively to show worker status:

$ adctrl interactive=n defaults_file=$APPL_TOP/admin/prod/ctrldefs.txt logfile=adctr.log menu_option=SHOW_STATUS

Using any menu option on the command line, except for SHOW_STATUS, requires that you also use the worker_ range=<range> option. See the AD Controller command line help for details.