Troubleshooting MDM Issues

At any step of MDM project development we face multiple issues. Troubleshooting those problems and fixing them is major task in MDM development. It is also time consuming process. However its not possible to have same solution for all problems , having systematic approach to resolve this problem can definitely help to solve in better way and faster. This are few things I follow while trying to fix issue in MDM:

Steps to troubleshoot :

1. View the logs to get the exact error :

Sometimes you may see that logs does not contain the error which you are facing . This could be either logs are rolled out or you might be checking different logs. You can either increase log size or write log file content into new file to track all the activity. You can change log mode to debug mode to get more details and stack trace for the specific error.

2. Search related KB : 

You can search the exact error code in informatica KB portal. You will find multiple KB's for specific error. You can open those KB's and check which is most relevant KB for the error which can help to resolve the issue. Sometimes you can search caused by  statement in KB to get more relevant KB. 

3. Search Java and DB error code 

In case if you don't find any KB with given error code , analyze logs to get more details : error will contain java error code or db error code. You may google this error code to get more details of the error and possible solution to fix the issue.

4. Isolate the problem : 

Few issues happens only in specific condition or specific environment. Understanding what exact configuration change is causing this error, will help to resolve it.

5.Informatica support

 In case if you are still not able to resolve issue you can take help of Informatica support team by raising case.

Common Informatica MDM Issues :

1. Installation and Upgrade : 

Most error will be due to unsupported version of component , missing environment variables , permission issues   

2. Performance Issue : 

You can review logs to understand which step is taking more time. You may find some specific query or procedure which is taking time. Depending on which you can change parameters to improve performance.

3. Job Failure : 

It could be because of connectivity issue or specific to data. Understanding data flow in job can help to troubleshoot this.

4. Configuration Error :

If you have configured any new thing and it does not work as expected or getting any error while execution , you can recheck if you have followed all steps as per given in the document. Any missing step can cause the issue . Going back and checking all steps will help. 

My secret  : Most of the time having small tea break has helped me to get better solution. Yes, it just breaks thought process which was in loop and gives me new direction or approach to look into it. I think you must try it  😃

 

Security Basics: Key Elements You Need to Know

 

Master data management is a crucial system for managing golden records effectively. Therefore making sure that this information used by right person , by right way is equally important. Who can use data , what data should be accessible to user , and how to use it effectively are few important questions which can be addressed by different security modules of MDM.

Tool Access: Tool access is available in hub console and it helps to manage available process and workbench. User need to have Administrator access to work with "Tool Access " .

Dynamic Data Masking : This feature helps to mask sensitive information when user opens any record. Dynamic Data Masking is Informatica product. So you need to make sure that you are using supported version of Dynamic data masking.

Security Access Manager : This tool is accessible from MDM hub console. This will protect unauthorized access to MDM resources which are like Base Object , Cleanse function. You can assign below privileges :

1. Read 
2. Create
3. Update
4. Delete
5. Merge
6. Execute

Authentication : Authentication is the process of identifying the user. User name and password will help to authenticate. Type of Authentication :

1. Internal : User will be created in MDM hub and will be having password stored into MDM.
2. External : User will be authenticated using external system like LDAP , Microsoft Active Directory.

Authorization : Authorization is the process to determine if user has sufficient privileges to perform operation which user is trying to do. 

Secure Resources and Privileges : Resources can be secure or private. MDM does have below resources : 

Base objects
Mappings
Packages
Cleanse functions
Match rule sets
Metadata
Profiles
Users table 

We can assign privileges to this resources. You can create group of resources and assign to roles.

Roles :  In MDM implementation, a variety of roles play crucial parts, including project manager, architect, developer, and administrator. To effectively manage MDM data, key roles such as business analysts, data stewards, and business users are essential. Furthermore, users have the flexibility to create new roles tailored to specific business requirements

Data Matching

Data matching is process to identify to similar or identical entities to merge and create single view of those entities. Data will be collected from multiple sources and there can be entities which are identical. Find those entities from different sources by comparing attributes of this entity is data matching process.

To understand this let’s take example of bank database where we can have customer data. This data will be collected from various sources e.g. loan department, saving account department, insurance department. There will be chances that single customer is part of this three department. Based on attribute like address, phone, name and email we can identify identical customer. This process is called data matching.

Data matching can be done by two ways :

1. Fuzzy matching :  Fuzzy match technique is used to find relevant data set. It identify two words identical based on phonetic or based on similarity between pronunciation. Example : Monika and Monica can be considered as match
2. Exact matching :  As name suggest if two records are copy of each other they will be consider exact match. Example : Monika and Monica will not be considered as match. Only Monika with another same word Monika will be matched. 

Benefits of Data Matching :

• Unnecessary costs reduced : Lets consider if bank is sending speed post every month to the customer. Customer has changed address into one system but did not update to speed post system . Bank can identify the customer and can send speed post to new address. It also helps in data storage reduction.
• Identifying duplicate records :  Record will be collected from various system matching will help to identify duplicate records
• Verifying the accuracy of data : If we find two duplicate records we can compare them to get accurate consolidated record
• Consolidating data : we can create consolidated record by collecting record from multiple sources which is best version if record.

Challenges with data matching :

• Data entry issues : While entering data into system someone can use different spelling for same name or they can use some short names. This could be Incorrect or incomplete data.
• Mismatches in data formats : Best example of mismatch in data format is phone number.          (91) 12345678 and 9112345678 this both are same however due to different formats this will not get detected. We can use data standardization technique to avoid this.
• Overmatching and undermatching : If matching rules defined are loose then we can get multiple records which can be overmatching. If matching rules are too tight we will get very less number of duplicates this will be undermatching.

Use case :

• Retail Sector : Retail sector can use MDM to get customer data and based on customer purchase behavior they can provide some offers to specific customer
• Financial Services : Bank can use this technique to identify potential fraud .Before giving any loan bank can check if this is existing customer ,if he has already taken any loan in the bank.
• Healthcare Sector : Healthcare Sector can use this technique to compare patient with his medical history so that before treating patient they will be aware of his existing health situation.
• Marketing and Sales : Marketing and Sales team can use this information to plan any new marketing scheme where they can compare customer purchase history and design offers to the customer.

What is Hotspot in MDM Match

If the record has too many matches those records are considered as hotspot. Depending on match configuration sometimes too many matches will be generated and those can be irrelevant matches.

 

Here are few reasons that can cause hotspot:

• Noise data :Noise words are common words and does not get identified by there own. Ideally this data will be removed by match process but if there is any word which is commonly used in the records but not recognized in match process , it can cause hotspot. Example: If Source is sending customer name by adding word "INACTIVE" to indicate inactive customers , a large number of records in MDM system will have customer name with "INACTIVE" word , potentially leading to irrelevant matches. 
• Search Level : Exhaustive and Extreme search level can cause overmatching.

Impact of Hotspot:

1.  It causes delay in the match process
2.  Sometimes match job will be failed with timeout
3.  Irrelevant records will be matched

How to handle hotspot:

This are few solutions we can try:

• DMAT : Dynamic match analysis threshold is used in match process to limit number of comparisons in match process.
• Match Keys Distribution : This tool in match configuration will help in identifying hotspot.
• During match process you can keep this records on hold with consolidation_ind=9. After match and merge you can manually review this records.
• EXCLUDE_FROM_MATCH : You can create column in BO with name "EXCLUDE_FROM_MATCH" and you set this value to 1 for the records which are causing overmatch.
• Cleansing and standardization: Proper cleansing and standardization helps to reduce noise data.
• Change in Match Configuration : Adding at least one exact column in match rules and selecting correct search level can help in reducing overmatch.

Multiple ways to execute LOAD Job in MDM

Loading data into a base object is known as a load job. All Master Data Management operations to obtain Golden record will be performed on this base object record. Here are the different possible ways to load data into base object or to call load job:

1. Batch Viewer : 

            You can execute MDM load job directly from hub console . In hub console under batch viewer tab you will get list of base object. Select base object and perform load Job.

• user should have hub console access
• batch job metrics will be accessible easily
• Job History will be visible to user
• Job scheduling is not possible

2. SIF API :

           Informatica provides SIF API ExecuteBatchLoad to call load job. This API can be called from SOAP UI or by using java EJB call. You can call SIF API from SOAP to load data into MDM.

• User does not require to login hub console
• It does not provide user interface to get job metrics and job history
• API can be called from java class by creating EJB client  

Sample Request Parameter:

username - user who is executing this load job

password - password of the user

orsId - databse ID 

tableName - stage table name

forceUpdateInd - 1 or 0  ( if the value is 1 MDM loads data regardless of last update date and if 0 , MDM only loads data that has more recent last update date)

Sample ResponseParameter:

message : Contains a message regarding status of the job

retCode : Provide return code

3. Command Line Batch Execution:

MDM batch load can be performed by command line execution. This can be done using The Command Line Batch Execution Resource Kit.   

-username -password action -tablename [-forceload]

• Command Line Batch Execution internally calls SIF API
• User does not require hub console access
• User can call this job from any scheduler tool 
• Can be used in shell scripts

If you know of any other ways to execute a load job, we'd love to hear from you in the comments section

Informatica MDM Core Components

 

Here is the list of MDM core components and how they are used in Multidomain MDM architecture

 

1. MDM hub master database (CMX_SYSTEM) 

This is master database which holds information about ORS database, MDM users, message queue settings , IDD application configuration.

The default name of MDM hub master database is CMX_SYSTEM but you can use custom name.

2. Operational Reference Store (CMX_ORS)

CMX_ORS stores business data, content metadata and rules to process and manage the master data. It will store all Base object, stage table ,landing table and REPOS tables which will have configuration details.

3. Hub Server

Hub Server is J2EE application that can be deployed to any of the application server supported by MDM. There are three application servers supported by MDM :

•     JBoss
•    WebSphere
•     WebLogic

 It manages core and common MDM services.

4. Process Server

Process Server processes batch jobs such as load, cleanse , recalculate BVT ,match & merge. It is essential to register the Process Server from the hub console after successful deployment to the application server.

5. Provisioning Tool

Provisioning tool will be used for creating and designing e360 portal. You can create multiple applications in provisioning tool and publish to the MDM .

6. ActiveVOS Schema

ActiveVOS schema will be used to store ActiveVOS metadata and task details.

7. Informatica ActiveVOS

This is business process management tool which helps to automate business process and task. By default Informatica provides predefined MDM workflows however you can customize this workflows using  ActiveVOS Designer.

8. Informatica Data Director

IDD is user interface to create , update and manage master data record. This will create real time records into MDM.