Identity & Access Management Learning: BulkLoad

Based on my experience with OIM Bulk Load Utility, below are the common issues/error one might encounter and how to solve/fix them:

Bulk Load Utility Does Not Handle Account Load With Employee Number In Recon Rule

When running the account data bulk load for one of the resource which has reconciliation rule based on Employee Number, the load failed with the following error
Exception in thread "main" java.sql.SQLException: ORA-00904: "null": invalid identifier
ORA-06512: at "DEV_OIM.OIM_BLKLD_PKG_ACCOUNT_LOAD", line 1562
ORA-06512: at line 1
at oracle.jdbc.driver.SQLStateMapping.newSQLException(SQLStateMapping.java:74)
at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:131)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:204)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:455)
......

Solution:

This is a known Bug# 17357095 and is fixed in OIM 11g R2 PS1 BP07 as per oracle. However, what i noticed that in some of the environment, the issue got resolved after applying patch and in some environment, the issue persists even after applying patch. So, as a workaround, I added the UserID/Login field was added in the Process Form, Resource Object and reconciliation rule was build using the UserID/Login. The field User ID was set with property Visible as false. Also, while generating the input account data file, I make sure that it contains use the UserID/Login field.

Bulk Load Utility throwing Null Pointer Exception while Trying To Load Account Data

When trying to bulk load account data with privileges and roles, having the following error after inputting the table space name to be used for creating temp tables :

Exception in thread "main" java.lang.NullPointerException
at bulkload.AccountLoad.createTmpTableNames(AccountLoad.java:857)
at bulkload.AccountLoad.inputCSV(AccountLoad.java:502)
at bulkload.AccountLoad.showSubMenu(AccountLoad.java:138)
at bulkload.AccountLoad.main(AccountLoad.java:1160)

Don't see any other error in the log file, and the utility just stops.

Solution:

The issue is encountered when the column in the input file doesn’t start with UD_ prefix. For example, the file with below header will see this issue. So, review your file header and fix the column name and retry the bulk load.

Bulk Load Of Account Data When Child Tables Need Not Be Populated

While using OIM Bulk Load utility for Loading Account Data( For examplefor OID User (UD_OID_USR) resource which doesn’t have any data in the child table UD_OID_GRP), specifying an input file for just the parent table does not work. The oim bulk loader forces the user to pass file names for the child tables as well. Output from the bulk loader is as below:

**********************************
Select the input for account load:
**********************************
1) DB Table
2) CSV File
3) Exit

Enter your option (1, 2 or 3):
2

Enter the resource object name:
OID User

Enter the comma separated CSV file names (parent CSV followed by child CSVs):
UD_OID_USR.csv

ERROR ==> The number of CSV files provided as input does not match with the number of account tables.

Exiting Utility ....

Solution:

Even if there is no data in the child table, you have to create files for both the parent & child tables and specify them during the bulk load utility.

Bulk Load Utility throwing ORA-01502 index or partition of such index is in unusable state

While loading the user/account data in OIM using bulk load utility, below error is encountered:

Exception in thread "main" java.sql.SQLException: ORA-01502: index 'HA_OIM.IDX_OIU_ORC_KEY' or partition of such index is in unusable state
ORA-06512: at "HA_OIM.OIM_SP_MANAGEENTITLEMENT", line 81
ORA-06512: at "HA_OIM.UD_PSHCM_R_ENT_TRG", line 4
ORA-04088: error during execution of trigger 'HA_OIM.UD_PSHCM_R_ENT_TRG'
ORA-06512: at "HA_OIM.OIM_BLKLD_PKG_ACCOUNT_LOAD", line 1847
ORA-06512: at line 1

Solution:

Bulk Load Utility tries to access an index or index partition i.e., 'HA_OIM.IDX_OIU_ORC_KEY' that has been marked unusable.

DROP the specified index, or REBUILD the specified index, or REBUILD the unusable index partition. Here are the queries DBA have been you using to do this:

Procedure to verify and Rebuild indexes:

1. Verify Indexes :
select * from dba_indexes WHERE status <> 'VALID' and owner in (‘HA_OIM’);

2. Create script to rebuild indexes online:
Spool rebuild_index.sql
SELECT 'alter index '||owner||'.'||index_name||' rebuild online ;' FROM DBA_INDEXES WHERE owner='HA_OIM' and status <> 'VALID' ;
Spool off;

3. Rebuild indexes:
Start rebuild_index.sql

Index rebuild for OIU Table:

• alter index HA_OIM.IDX_OIU_ORC_KEY REBUILD ONLINE ;
• alter index HA_OIM.IDX_OIU_USR_KEY REBUILD ONLINE ;
• alter index HA_OIM.IDX_OIU_OBI_KEY REBUILD ONLINE ;
• alter index HA_OIM.IDX_OIU_OST_KEY REBUILD ONLINE ;
• alter index HA_OIM.IDX_OIU_APP_INSTANCE_KEY REBUILD ONLINE ;
• alter index HA_OIM.FDX_OIU_ACCOUNT_TYPE REBUILD ONLINE ;
• alter index HA_OIM.IDX_OIU_REQ_KEY REBUILD ONLINE ;
• alter index HA_OIM.IDX_USG_USR_KEY rebuild online ;
• alter index HA_OIM.IDX_USG_UGP_KEY_USG_PRIORITY rebuild online ;

Creating Indexes Online:

CREATE INDEX HA_OIM.IDX_OIU_ORC_KEY ON HA_OIM.OIU
(ORC_KEY)
LOGGING
TABLESPACE HA_OIM
PCTFREE    10
INITRANS   2
MAXTRANS   255
STORAGE    (
            INITIAL          64K
            NEXT             1M
            MINEXTENTS       1
            MAXEXTENTS       UNLIMITED
            PCTINCREASE      0
            BUFFER_POOL      DEFAULT
            FLASH_CACHE      DEFAULT
            CELL_FLASH_CACHE DEFAULT
           )
NOPARALLEL online;

Note: I faced this issue only in production environment and it seems to be coming for every bulk load operation I did. I also noticed this issue I try to rename the logs folder before the bulk load utility complete all its operation, I mean enabling and re-building indexes.

Addendum on this issue:

ORA-08104: this index object HA_OIM.IDX_OIU_ORC_KEY is being online built or rebuilt

Due to missing index on ORC_KEY, the below sql statement was causing performance issue on the data load operation:

SELECT OST.OST_KEY, OIU.OIU_KEY, OIU.USR_KEY
FROM OST OST, OIU OIU
WHERE OIU.OST_KEY = OST.OST_KEY AND OIU.ORC_KEY = :B1

After the data load got complete for one of the resource, DBA ran the dbms_repair.online_index_clean function.

If you have been using OIM Bulk Load Utility to load the accounts,users,roles and et al, a piece of advice, please make sure that you keep the input file name to less than 25 characters else you will see below error in the logs:

***************************************************************************************************
Processing File: Users_DisabledUntilStartDate.csv
---------------------------------------------------------
java.sql.SQLException: ORA-01400: cannot insert NULL into ("HA_OIM"."OIM_BLKLD_LOG"."LOAD_SOURCE")
ORA-06512: at "DEV_OIM.OIM_BLKLD_SP_LOG_MSG", line 21
ORA-06512: at "DEV_OIM.OIM_BLKLD_PKG_USR", line 1505
ORA-06502: PL/SQL: numeric or value error: character string buffer too small
ORA-06512: at line 1
        at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:445)
        at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396)
        at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:879)
        at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:450)
        at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)
        at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531)
        at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:204)
        at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:1034)
        at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1329)
        at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3584)
        at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3685)
        at oracle.jdbc.driver.OracleCallableStatement.execute(OracleCallableStatement.java:4694)
        at oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1088)
        at bulkload.UserLoad.processData(UserLoad.java:373)
        at bulkload.UserLoad.loopAllCSVs(UserLoad.java:264)
        at bulkload.UserLoad.inputCSV(UserLoad.java:172)
        at bulkload.UserLoad.showSubMenu(UserLoad.java:95)
        at bulkload.UserLoad.main(UserLoad.java:988)
---------------------------------------------------------

===============================================================================
Time taken in re-building indexes and enabling FK constraints
===============================================================================
Start time:      24-JUL-14 05.57.11.971759 PM
End Time:        24-JUL-14 05.57.35.589116 PM
Seeing this error in the logs, my first thought that something might be wrong with my input file but after rounds of verifying the column name and data, I decided to ask google about this error. I was lucky enough to find time some pointers quickly and finally figured it out that problem was with length of input file. After changing the file name to lo have less than 25 characters, the load went fine.

Below is the forum post that provided me pointers on this issue:

https://community.oracle.com/thread/1062129

Identity & Access Management Learning

Thursday, September 25, 2014

OIM11gR2 : Common Bulk Load Issues

Sunday, July 27, 2014

OIM11gR2: Bulk Load Bugs