Driver Jobs – Better Than Magic

Have you ever found yourself needing your NetIQ IDM solution to perform a set of instructions at a specific time of day or at regular intervals? Most solutions for IDM include some timed processes like nightly checks for upcoming password or account expirations that require email notifications to account holders or managers. The challenge most people face with these types of processes is understanding how to execute a regularly scheduled routine in the NetIQ’s real-time, event triggered IDM system. Luckily NetIQ has already thought about that too and has given us more than one way to achieve this.

In this blog I will talk about the first method, driver jobs.

What is a driver job?

Well that is the most obvious question and basically a driver job is a command scheduled in a driver’s configuration to automatically trigger an event in that driver at a specified time or interval. Now, this is not to be confused with a pre-programmed set of instructions that are automatically executed. In fact, a driver job by itself executes no policies or rules within a driver. All a driver job will do is raise an event within the driver’s engine that can then be detected by rules to trigger whatever actions are required.

That still doesn’t make sense, can you explain more?

So let’s look at this in the terms of a driver rule. In a rule you have conditions and actions. When an event is raised within eDirectory it triggers the driver to evaluate the transaction to determine what rules are executed and what rules are ignored based on each rule’s conditions. Typically you have a rule condition that says “if class name is equal to ‘User'” and if true then that rule executes the actions defined within.

Well, a driver job basically raises an event within that driver that only includes the name of the job. Nothing else. No object attributes or anything else; just the job’s name. It is up to the rule conditions within the driver to determine if that rule needs to be executed for that job. You do not declare all of your logic in the job configuration.

Alright. So a driver job is just a job name and scheduled time but how do I create jobs for my drivers?

Like with other driver tasks, jobs can be configured and deployed through Designer or configured directly through iManager.

Note: It is the preferred practice to do all development and configuration in Designer that is then deployed to eDirectory. It is not recommended to perform standard development tasks through iManager, especially in environments where there are multiple people managing/developing for the IDM solution to minimize Designer synchronization conflicts that may cause changes to be overwritten or deleted during subsequent Designer deployments. Because Designer is the preferred development interface this post will focus on how to create/manage driver jobs through Designer only.

Once you have your Designer project open just right click on the driver you would like to create the job for in the Outline tab view and select “New” then “Job” in the context menu that appears.

b2ap3_thumbnail_NewJob

In the New Job window that appears just enter a unique name for the job to be created in the “Name” field, select the “Subscriber channel trigger” job definition and then select which server(s) in your IDM solution this job will run on. Once that data is entered/selected click the OK button to create the job.

Note: There should be a checkbox that is selected by default that opens the job for editing after it is created. It is recommended that you leave this checkbox selected but if for any reason you need to edit this job later it will appear under that driver in the Designer Outline tab where if you double-click the job object the editor will open.

b2ap3_thumbnail_JobName

In your screenshot under the Installed selection there are multiple Job Definitions items listed. Do drivers have preconfigured jobs that can be enabled?

Yes, there are some preconfigured job types that can be selected. The “Random Password Generator” job should be pretty straightforward on what it does. The “Schedule driver” job allows you to schedule an automated stop or start of the driver service. The “Subscriber channel trigger” job is the one that is most often used however and is the focus of this post. This job type generates a generic event to the driver that allows us to create rules to perform whatever actions are required.

So once your job is created and the editor open you will notice that it looks similar to other object editors with multiple tabs that allow you to access different sections of its configuration.

b2ap3_thumbnail_Editor

Just leave the information in the General tab where it is. There is no need to add a scope for most jobs under this type. Instead, click the Job Parameters tab and change the first option for “Submit a trigger document for objects without a driver association” from “false” to “true”. Because the job is submitting a zero-trigger document without a scope we need this option for the blank trigger to be properly recognized.

b2ap3_thumbnail_Parameters

In most cases it is recommended that the “Method for submitting trigger documents” to be left to the default value of “queue (use cache)”. This option allows the driver to continue processing any cached transactions that are in the TAO files already before executing the job. In layman’s terms, this option just puts the job at the back of the line if there is one and it waits its turn just like any normal transaction within the IDM system. If you have a need to have your job supersede any pending transactions change this option to “direct (bypass cache)” to have the job placed in the front of the line so that it is processed next regardless of what else may be in the driver queue.

Note: The queue option is generally recommended so that the job acts on the most up-to-date data in the directory at the time of its scheduled execution. If you have multiple batch jobs that are executed against your eDirectory (like nightly HR dumps) it is generally preferred to have this data processed before running jobs that may need that data or data related to it.

Once you have set the desired parameters click the Schedule tab. This tab is pretty obvious but this is where you can select your options to schedule how frequently this job is triggered within your environment.

b2ap3_thumbnail_Schedule

In this tab you can configure to have the job run daily at the desired time resulting in the job being triggered automatically once every day, 7 days a week, 365 days a year for as long as the driver is running. If you don’t want every day of the week then you have the option to choose which specific days of the week the job will be triggered every week for as long as the driver is running.

Still too frequent? If you just need your job to run on specific days of the month (like the 15th & 30th) then choose the Monthly option and click the plus sign ( + ) to be given a list of days to select. The job will only be executed on those days each month for as long as the driver is running. Just be careful using this option because if you select 31 for the end of the month but a month only has 30 days the job will not be executed for that month.

Need it to run less often than that? If you only need your task to in specific months or even on specific days of specific months then choose the yearly option where you can choose which specific days and months the job will be needed to run. And just like the Monthly option this option can be tricky. The job will be executed for each day selected in each month selected. You cannot use this option to trigger a job on April 15th and June 1st. You would have to have select April & June as the months and 1 & 15 for the days so this job would run 4 times; April 1st, April 15th, June 1st and June 15th.

Still not enough control? Rest easy because if you need to create custom, complex scheduling rules for this job use the Custom option to create a custom crontab command that meets your needs. Be careful using this option though because if the command is improperly configured it could result in the job not being triggered at all.

With the job configured save your changes and deploy the job from Designer to eDirectory just like you would with anything else like a driver entitlement or policy. The key thing to remember during this process is that jobs require rights to execute and inherits the rights of the parent driver. If you have any issues running the job check the “security equals” setting on the driver to make sure the job has sufficient rights to the directory to inject the trigger document. Most drivers generally run with privileges equal to an administrator so if your driver is running with restricted or reduced directory rights this may require some adjustment to the driver’s permissions.

Well that seems too simple. How does this allow me to execute driver rules?

Well the job is only the first step. While the job doesn’t perform any real processing in itself it does create a trigger that signals the driver to take action. There are some other components that are needed within the driver and then all of these components work together to achieve the overall desired goal. In fact, in order for the driver to take action on the job trigger you will need to create at least one rule within your driver to detect the trigger and then perform standard rule actions when that condition is met and how to create a rule that detects that trigger is our next topic.

To take advantage of the job configured create a new rule, preferably in the Event Transform policy set of the Subscriber channel. The new rule only needs two conditions:

if operation equal “trigger”
if operation property ‘source’ equal “
The first condition just checks to see what operation type the document is. Trigger is an operation type similar to add, modify, delete, rename, move and the other common operations drivers detect. The second just adds a second level of filtering so that the rule only acts on this job. In a driver where there is only one job configured this rule isn’t necessary as the driver will only have one trigger but it is a good practice to always include this condition to reduce future risk and minimize potential future work if the driver is ever expanded to include multiple jobs.

b2ap3_thumbnail_RuleConditions

From here you would just define your actions as normal to do whatever was needed for that trigger.

But I need to query eDirectory to get a subset of my users that meet a specific set of criteria as part of this trigger. How can I use this to perform a set of actions for a filtered group of users within my directory?

This is the most common need for these types of jobs; a daily job that searches for users that meet some criteria, usually a pending expiration of some type, and then generate an email to that user and/or that user’s manager to notify of the upcoming deadline or expiration. And for that there are two solutions.

The old school method, which still works in new IDM environments, is to create an EcmaScript on the driver to perform the desired LDAP query against eDirectory (or any other LDAP compatible directory) that returns the results in a node set array. With the EcmaScript created just call it as part of a “for each” loop in the driver actions to iterate through the result set and perform the desired actions for each object returned.

b2ap3_thumbnail_ForEach

In the new versions of IDM and Designer there is a Query noun in the argument builder that lets you define a custom LDAP query that also returns a node set array that can be used with a for each loop.

b2ap3_thumbnail_Query

And that is really all there is to it. I know it may sound a bit complex and it took a lengthy post to describe it but really all it requires is a driver rule to detect a trigger that acts on the job and the actual scheduled job. Once you are familiar with the process you can create a job in just a few minutes and then depending on the complexity of what needs to be done as part of that job the rule development can vary in length but for an experienced driver developer the rule shouldn’t take more than a few hours.

What if I need to run my job as part of a “one-off” or something? Can I manually run my jobs?

Yes! Even though driver jobs are intended to execute as scheduled tasks you can execute jobs at any time through iManager. When inspecting the driver hosting the job there is a Jobs tab and under that tab you can select the job(s) you want to run and then click the “Run Now” link to manually start the selected job(s).

However, there are a few things to keep in mind while considering if or how to implement scheduled jobs that we need to discuss.

If you plan on creating a job that could potentially act upon hundreds of objects or more within your directory you may not want to have that job scheduled on a driver that is responsible for processing a lot of real-time data. For example, if you want a job that sends emails to users who have account expiration dates within the next 5 days you probably don’t want to put that job on a driver that constantly performs provisioning/de-provisioning processes like an HR, Active Directory or Exchange driver. The simple reason is that these jobs can take several minutes to hours to complete depending on the number of objects in the queries node set and the various actions performed for each object found.

Also, when considering what driver to put jobs on I would also recommend that you not put lengthy jobs on a driver that may not be responsible for provisioning accounts but provides provisioning support. For example, many NetIQ IDM implementations leverage a Loopback driver that processes data as it changes within eDirectory to determine access rights or sets other attributes in eDirectory based on new or changing values (like full name, account enabled, etc.). Putting lengthy jobs on these drivers could also negatively impact provisioning/de-provisioning times that could result in accounts not being created or disabled for hours after the expected times. In most systems where there are lengthy jobs to be executed regularly it is generally recommended that a new Loopback/Null driver be created and that driver’s only responsibility is to host/execute those jobs so that whatever processing time is required does not impact the rest of the system’s ability to provision/de-provision/maintain accounts across the enterprise.

If you plan on having multiple jobs on a single driver try to avoid having them triggered at the same time. The impacts of this should be obvious but it is generally recommended that you try to space out your jobs schedules. Execution times may vary from day to day depending on the data processed in each execution but try to estimate the average run time and schedule accordingly just for ease of understanding and troubleshooting.

If a driver is executing a for each loop it will not shut down or stop until the loop is completed. This is important to understand if a job executes that processes several records as a batch job. This means that if you have a job that requires 30-60 minutes to run and you discover an error in the rule after 5 minutes that you will not be able to simply stop the driver through iManager or Designer to stop the process. If you attempt to stop or shut down the driver during this process the driver will indicate it is “shutting down” but stay in that state until the process completes. Repeated attempts to stop or shut down the driver will result in errors through the UI but will have no impact on the driver or the running process. If it is absolutely necessary to stop the process the only thing you can do is restart that eDirectory instance as a whole. By restarting eDirectory (or even the server itself) it will force the driver to stop and it will purge the transaction in process. This will allow you to make any corrections to the job or driver rules needed without impacting everyone. After any needed corrections are made you can manually start the job through iManager to ensure any processes needed are performed without having to wait until the next scheduled time.

But above all else, if you are using a newer version of NetIQ IDM there is a capability called Work Orders. Work Orders are objects in eDirectory that contain various pieces of information that include an execution date/time. The WorkOrder driver polls eDirectory looking for Work Order objects that are ready to be executed and then triggers an event in the driver similar to how a schedule job does. The key difference to this is that you can dynamically create/delete Work Order objects through driver rules that target specific users. Work Orders are commonly used for performing common actions but individual users. For example, all new users need to have Exchange mailboxes created but due to Active Directory replication times you don’t want to provision the Exchange account until 15 minutes after the AD account is provisioned. In this scenario you have a Work Order driver that has a rule to grant an Exchange role/resource/entitlement and during the AD account provisioning process the AD driver creates a Work Order set to execute 15 minutes later and includes the new user’s DN as part of the Work Orders information. 15 minutes later when the Work Order is executed the rule pulls the target user’s DN from the Work Order and grants the necessary data to result in an Exchange mailbox being provisioned. (A future blog posting will cover work orders in more detail.)