#infotrellis

Web Services Overview: Web Services are services available over the web that enables communication and provide a standard protocol for communication. To enable the communication, we need a medium (HTTP) and a format (XML/JSON).

There are two parties to the web services, namely Service Provider and Service Consumer. A web service provider develops/implements the application (web service) and makes it available over the internet (web).  Service Provider publishes an interface for the web services that describes all the attributes of the web service. Service Consumer consumes the web service. For the Consumer to consume the web service, the consumer has to know the services available, request and response parameters, how to call the services and so on.

Hence we can define Web Service as a standardized way of integrating web desk applications using XML, SOAP, WSDL and UDDI open standards over an internet protocol backbone. XML is used to tag the data. SOAP is used to transfer the data. WSDL is used for describing the services available and UDDI is used for listing what services are available.

Banking Regulations – Overview

Managing regulatory issues and risk has never been so complex. Regulatory expectations continue to rise with increased emphasis on the institution’s ability to respond to the next potential crisis. Financial Institutions continue to face challenges implementing a comprehensive enterprise-wide governance program that meets all current and future regulatory expectations. There has been a phenomenal rise in expectations related to data quality, risk analytics and regulatory reporting.

Following are some of the US regulations that MDM and customer 360 reports can be used for compliance:

FATCA (Foreign Account Tax Compliance Act)

FATCA was enacted to target non-compliance by U.S. taxpayers using foreign accounts. The objective of FATCA is the reporting of foreign financial assets. The ability to align all key stakeholders, including operations, technology, risk, legal, and tax, is critical to successfully comply with FATCA.

OFAC (Office of Foreign Asset Control)

The Office of Foreign Assets Control (OFAC) administers a series of laws that impose economic sanctions against hostile targets to further U.S. foreign policy and national security objectives. The bank regulatory agencies should cooperate in ensuring financial institutions comply with the Regulations.

FACTA (Fair and Accurate Credit Transactions Act)

Its primary purpose is to reduce the risk of identity theft by regulating how consumer account information (such as Social Security numbers) is handled.

HMDA (Home Mortgage Disclosure Act)

This Act requires financial institutions to provide mortgage data to the public. HMDA data is used to identify probable housing discrimination in various ways.

Dodd Frank Regulations

The primary goal of the Dodd-Frank Wall Street Reform and Consumer Protection Act was to increase financial stability. This law places major regulations in the financial industry.

Basel III

A wide sweeping international set of regulations that many US banks must adhere to is Basel III. Basel III is a comprehensive set of reform measures, developed by the Basel Committee on Banking Supervision, to strengthen the regulation, supervision and risk management of the banking sector.

What do banks need to meet regulatory requirements?

Organizations spend millions of dollars to implement their MDM solution. They may have different approaches (batch vs. real time; integrated customer view vs. integrated supplier view etc.) – but in general they all expect to get a “one version of the truth” view by integrating different data sources and then providing that integrated view to a variety of different users.

After the completion and successful testing of the MDM implementation project, companies sit back and enjoy the benefits of their MDM hub – and more often than not don’t even think about looking under the hood. It never occurs to them that they could be trying to gain insights into what’s happening inside that MDM hub by asking questions like

–          How is the data quality changing?

–          What are the primary activities (in processing time) inside the MDM hub?

–          How are service levels changing?

However, organizations change, people change, requirements change – impacting what is happening inside the MDM Hub. Such changes can open up significant opportunities for an organization – but without doing any sort of investigation that opportunity is typically not recognized.

Here are two examples – diagnosed through the use of an MDM audit tool:

–          The company’s MDM Hub had approximately 100,000 incorrect customer addresses. These addresses were used for regular mailings; the mailings generated (in case of correct address) incremental revenues. Impact on the business related to just one mailing:

$400K wasted on the mailing cost ($4 is the conservative mailing cost per person – for postage, printing of the mailer etc.) $100K of immediately lost revenues (as past data shows that one in 50 customers spends about $50 immediately following the mailing)

The longer term revenue lost was not assessed, but was estimated to be well over $400K

The opportunity: Cost saving of $400k and revenue increase of $500K or more –          At a different company, by analyzing data processed by week the resulting report was able to determine that the number of new customers processed was declining by 1-2% every week – starting about 6 weeks before the audit was conducted. A deeper review of the audit report suggested that

The original service levels related to customer file changes had been getting worse and worse over that same time period

As customer file changes (as per the audit report) took over 85% of the total processing time, the slower processing lead to less time available for new customer processing

This initial diagnostic was confirmed by the client – they had a slowly growing backlog of new customer files

Ultimately the audit was able to highlight which input data source had been causing the slowdown, allowing the company to resolve the problem at its source

Business impact: a major risk (very significant slowdown in new customer set up) was eliminated before it became a real problem.

Introduction

MongoDB is an open-source document- oriented schema-less database system. It does not organize the data using rules of a classical relational data model. Unlike other relational databases where data is stored in columns and rows, MongoDB is built on the architecture of collections and documents. One collection holds different documents and functions. Data is stored in the form of JSON style documents. MongoDB supports dynamic queries on documents using a document based query language like SQL.

This blog post explains how MongoDB can be integrated with IBM DataStage with an illustration.

Why MongoDB? For the past two decades we have been using Relational Database as data store as they were the only option that was available. But with the introduction of NoSQL, we have more options based on the requirement. Mongo DB is predominantly used in insurance and travel industry.

We can extract any semi-structured data and load it to MongoDB through any of the integration tools. Also Extract from MongoDB is easier and faster when compared to relational databases.

MongoDB integration with IBM DataStage Since we don’t have a specific external stage in IBM DataStage tool to integrate MongoDB, we are going with Java Integration stage to load or extract data from MongoDB.

Since MongoDB is a schema free database, we can use structured or semi-structured data extracted through DataStage and load it to MongoDB.

Prerequisites

Make sure you have java installed on your machine.

Install Eclipse tool.

Java requires below MongoDB jar to be imported inside the package to use MongoDB functions

mongo-java-driver-2.11.3.jar or higher versions if available (Download it from the internet)

Also, Java requires below jar file to be imported inside the package to extract or load data from DataStage

jar (It is available on the DataStage server. Location: /opt/IBM/InformationServer/Server/DSEngine/java/lib)

Illustration of a DataStage job Create a job in DataStage to parse the below sample XML

IBM InfoSphere Master Data Management Collaborative Edition provides a highly scalable, enterprise Product Information Management (PIM) solution that creates a golden copy of products and becomes trusted system of record for all product related information.

Performance is critical for any successful MDM solution which involves complex design and architecture. Performance issues become impedance for smooth functioning of an application, thus obstructing the business to get the best out of it. Periodic profiling and optimizing the application based on the findings is vital for a seamless application.

InfoTrellis has been providing services in PIM space over a decade now for our esteemed clientele that is spread across the globe.

This blog post details on optimizing IBM InfoSphere MDM Collaborative Edition application based on the tacit knowledge acquired from implementations and upgrades carried out over the years.

Performance is paramount Performance is one of the imperative factors that make an application more reliable. Application performance of MDM Collaborative Edition is influenced by various factors such as solution design, implementation, infrastructure, data volume, DB configurations, WebSphere setup, application version, and so on. These factors play a huge role in affecting business either positively or otherwise. Besides, even in a carefully designed and implemented MDM CE solution, performance issues creep up over a period of time owing to miscellaneous reasons.

Performance Diagnosis The following questions might help you to narrow down a performance problem to a specific component.

What exactly is slow – Only a specific component or general slowness which affects all UI interactions or scheduled jobs?

When did the problem manifest?

Did performance degrade over time or was there an abrupt change in performance after a certain event?

Answers to the above queries may not be the panacea but provide a good starting point to improve the performance.

Hardware Sizing and Tuning Infrastructure for the MDM CE application is the foundation on top of which lays the superstructure.

IBM recommends a hardware configuration for a standard MDM CE Production server. But then, that is just a pointer towards the right direction and MDM CE Infrastructure Architects should take it with a pinch of salt.

Some of the common areas which could be investigated to tackle performance bottlenecks are:

Ensuring availability of physical memory (RAM) so no or little memory swapping and paging occurs.

Latency and bandwidth between the application server boxes and database server. This gains prominence if the Data Centers hosting these are far and away. Hosting Primary DB and App Servers on Data Center could help here.

Running MDM CE on dedicated set of boxes will greatly help so that all the hardware resources are up for grabs and isolating performance issues becomes a fairly simple process, of course, relatively.

Keeping an eye on disk reads, writes and queues. Any of these rising beyond dangerous levels is not a good sign.

Clustering and Load Balancing Clustering and Load balancing are two prevalent techniques used by applications to provide “High Availability and Scalability”.

Horizontal Clustering – Add more firepower to MDM CE Application by adding more Application Servers

Vertical Clustering – Add more MDM CE Services per App Server box by taking advantage of MDM CE configuration – like more Scheduler and AppServer services as necessary

Adding a Load Balancer, a software or hardware IP sprayer or IBM HTTP Server will greatly improve the Business user’s experiences with the MDM CE GUI application

Go for High Performance Network File System Typically clients go with NFS filesystem for MDM CE clustered environments as it is a freeware. For a highly concurrent MDM CE environment, opt for a commercial-grade, tailor-made high performance network file system like IBM Spectrum Scale .

Database Optimization The performance and reliability of MDM CE is highly dependent on a well-managed database. Databases are highly configurable and can be monitored to optimize performance by proactively resolving performance bottlenecks.

The following are the few ways to tweak database performance.

Optimize database lock wait, buffer pool sizes, table space mappings and memory parameters to meet the system performance requirements

Go with recommended configuration of a Production-class DB server for MDM CE Application

Keeping DB Server and Client and latest yet compatible versions to take advantage of bug fixes and optimizations

Ensuring database statistics are up to date. Database statistics can be collected manually by running the shell script from MDM CE located in $TOP/src/db/schema/util/analyze_schema.sh

Check memory allocation to make sure that there are no unnecessary disk reads.

Defragmenting on need-basis

Checking long running queries and optimizing query execution plans, indexing potential columns

Executing $TOP/bin/indexRegenerator.sh whenever the indexed attributes in MDM CE data model is modified

MDM CE Application Optimization

The Performance in MDM CE application can be overhauled at various components like data model, Server config., etc. We have covered the best practices that have to be followed in the application side.

Data Model and Initial Load

Carefully choose the number of Specs. Discard the attributes that will not be mastered, governed in MDM CE

Similarly, larger number of views, Attribute Collections, Items and attributes slower the user interface screen performance. Tabbed views should come handy here to tackle this.

Try to off-load cleansing and standardization activities outside of MDM solution

Workflow with a many steps can result in multiple problems ranging from an unmanageable user interface to very slow operations that manage and maintain the workflow, so it should be carefully designed.

MDM CE Services configuration

MDM CE application comprises of the following services which are highly configurable to provide optimal performance – Admin, App Server, Event Processor, Queue Manager, Workflow Engine and Scheduler.

All the above services can be fine-tuned through the following configuration files found within the application.

$TOP/bin/conf/ini – Allocate sufficient memory to the MDM CE Services here

$TOP/etc/default/common.properties – Configure connection pool size and polling interval for individual services here

Docstore Maintenance

Document Store is a placeholder for unstructured data in MDM CE – like logs, feed files and reports. Over a period of time the usage of Document Store grows exponentially, so are the obsolete files. The document store maintenance report shall be used to check document store size and purge documents that do not hold significance anymore.

Use the IBM® MDMPIM DocStore Volume Report and IBM MDMPIM DocStore Maintenance Report jobs to analyze the volume of DocStore and to clean up the documents beyond configured data retention period in IBM_MDMPIM_DocStore_Maintenance_Lookup lookup table.

Configure IBM_MDMPIM_DocStore_Maintenance_Lookup lookup table to configure data retention period for individual directories and action to be performed once that is elapsed – like Archive or Purge

Cleaning up Old Versions

MDM CE does versioning in two ways.

Implicit versioning

This occurs when the current version of an object is modified during the export or import process.

Explicit versioning

This kind of versioning occurs when you manually request a backup.

Older versions of items, performance profiles and job history need to be cleansed periodically to save load on DB server and application performance in turn.

Use the IBM MDMPIM Delete Old Versions Report and IBM MDMPIM Estimate Old Versions Report in scheduled fashion to estimate and clear out old entries respectively Configure IBM MDMPIM Data Maintenance Lookup lookup table to hold appropriate data retention period for Old Versions, Performance Profiles and Job History

Best Practices in Application Development

MDM CE presents couple of programming paradigms for application developers who are customizing the OOTB solution.

Scripting API – Proprietary scripting language which at runtime converts the scripts into java classes and run them in JVM. Follow the best practices documented here for better performance

Java API – Always prefer Java API over the Scripting API to yield better performance. Again, ensure the best practices documented here are diligently followed

If Java API is used for the MDM CE application development, or customization, then :

Use code analyzing tools like PMD, Findbung, SonarQube to have periodic checkpoints so that only the optimized code is shipped at all times

Use profiling tools like JProfiler, XRebel, YourKit or VisualVM to constantly monitor thread pools use, memory pools statistics, details about the frequency of garbage collection and so on. Using these tools during resource-intensive activities in MDM CE, like running heavyweight import or export jobs, will not just shed light on inner workings of JVM but offers cues on candidates for optimization

Cache Management

Keeping frequent accessed objects in cache is a primary technique to improvement performance. Cache hit percentage need to be really high for smooth functioning of the application.

Check the Cache hit percentage for various objects in the GUI menu System Administrator->Performance Info->Caches The $TOP/etc/default/mdm-ehcache-config.xml and $TOP/etc/default/mdm-cache-config.properties files can be configured to hold large number of entries in cache for better performance

Performance Profiling A successful performance testing will project most of the performance issues, which could be related to database, network, software, hardware etc. Establish a baseline, identify targets, and analyze use cases to make sure that the performance of the application holds good for long.

You should identify areas of solution that generally extends beyond normal range and few examples are large number of items, lots of searchable attributes, large number of lookup tables.

Frameworks such as JUnit, JMeter shall be used in a MDM CE engagement where Java API is the programming language of choice

About the author

Sruthi is a MDM Consultant at InfoTrellis and worked in multiple IBM MDM CE engagements. She has over 2 years of experience in technologies such as IBM Master Data Management Collaborative Edition and BPM.

Since 2007, InfoTrellis has been a pioneer in information management, helping clients realize full value from their investments in this space. InfoTrellis offers leading information management technology and expert consulting services to the world’s most recognized brands. The complete suite of products and services is highly-focused around Master Data Management, Big Data, Data Integration, Data Analytics and many more data-based processes enterprises use for Data Governance.

With rich experience in data management for more than a decade, InfoTrellis is pioneering big data management. Traditional data processing techniques are proving to be inadequate. We have the business acumen and technical expertise to provide the best-in-class solutions for handling big data.

Toronto, Ontario, Canada – July 7, 2017: InfoTrellis, Inc., a Toronto-based  Data Management & Analytics company, today announced that it has entered into a definitive agreement under which Mastech Digital, Inc. (NYSE: MHH), a leading Digital Transformation Services provider, will acquire InfoTrellis’ services division. The demand for Digital Transformation consulting services is growing rapidly, as more Fortune 1000 companies are funding digital transformation initiatives to reinvent how they do business.  Data & analytics is at the heart of every digital transformation strategy and our combination will bring together two differentiated companies – both with global brand recognition and complementary service offerings – and a unique value proposition to offer expertise and scalability in digital transformation consulting.

The acquired business, to be branded as Mastech InfoTrellis, will offer project-based consulting services to customers in the areas of Master Data Management, Data Integration, Big Data and Customer Intelligence Management (Customer 360), and at the same time strengthen Mastech Digital’s digital transformation services capability. InfoTrellis’ industry-wide recognition for its thought leadership and depth in data management & analytics, as well as proven global delivery expertise, will enhance the growth opportunities of the combined entity.

“Mastech Digital has global brand recognition and is highly respected in the IT industry, and InfoTrellis is very excited to join them,” said Mahmood Abbas, CEO and Co-founder of InfoTrellis.  “I am confident that our strengths in delivering consulting services in data management & analytics, combined with Mastech Digital’s scale and proven experience in providing IT staffing and digital transformation services, will offer a unique value proposition in the market of scalable consulting teams with deep expertise.”

The transaction is valued at USD $55 million, with USD $35.7 million paid in cash at closing and USD $19.3 million deferred over an earn-out period .

InfoTrellis and Mastech Digital expect to realize several synergies from the combined enterprise:

Ability to participate in larger digital transformation deals

Opportunity to cross-sell broader range of service offerings to existing and future customers of both businesses

Larger scale, improved operational efficiencies and expanded global delivery model

Better economies of scale from shared support services integration

“We began transforming Mastech into a digital transformation services company a year ago when we launched our new name ‘Mastech Digital’ and recast our service offerings,” said Vivek Gupta, President and CEO of Mastech Digital. “The acquisition of InfoTrellis will be a significant milestone in that journey. I am delighted with the capabilities InfoTrellis brings to the Mastech Digital family and am excited about creating a combined organization that will deliver world-class services around Data Management & Analytics in its expanded portfolio of digital transformation services.”

The combined entity will have augmented scale, expanded global delivery capability, and deeper leadership strength.  “The integration will advance our vision of addressing the wider challenges faced by our clients in adopting Big Data and Advanced Analytics,” said Sachin Wadhwa, COO and Co-founder of InfoTrellis.  “We believe the combination of Mastech Digital and InfoTrellis will have a meaningful impact in the way Data Management and Analytics consulting is delivered by us.“

Mahmood Abbas and Sachin Wadhwa will continue to play the leadership role in the combined entity.

The product division of InfoTrellis has been spun off as a separate company called AllSight, Inc., and Zahid Naeem, one of the Co-founders of InfoTrellis, has moved to the AllSight company. “Our vision for InfoTrellis consulting has always been to provide premium data management and analytics consulting services for our clients,” said Zahid Naeem.  “This partnership with Mastech Digital is a major milestone in continuing with that vision. We are also looking forward to a strong partnership between the combined entity and AllSight Inc., as Customer Intelligence Management, or a Customer 360, is a critical software system to deliver upon a digital transformation strategy.”  

InfoTrellis has over 200 associates globally, a customer-base of more than 40 blue chip customers in North America, offices in Toronto (Canada) and Austin (Texas, U.S.), as well as a global delivery center in Chennai (India).  Mastech Digital has nearly 1,300 associates globally, a customer-base of over 300 companies, seven offices across the US, as well as a delivery center in New Delhi (India).

The acquisition is subject to customary closing conditions and is expected to close in July 2017.

About Mastech Digital

Mastech Digital (NYSE MKT: MHH) is a national provider of IT services focused on solving its customers’ digital transformation challenges. The Company’s IT staffing services span across digital and mainstream technologies while its digital transformation services include Salesforce.com, SAP HANA, and digital learning services. A minority-owned enterprise, Mastech Digital is headquartered in Pittsburgh, PA with offices across the U.S. and India. For more information, visit www.mastechdigital.com.

About InfoTrellis

The digital era has fostered information transfer between two systems but the communication flaws have led to information loss. For example; Product Information shared between manufacturers and retailers. Manufacturers often communicate about new Products or changes to existing Products, Price information changes to retailers manually and in an ad hoc manner, leading to the data quality and integrity issues in key retail systems. These problems result in revenue loss and dissatisfied consumers.

Considering these challenges in mind, GDSN (Global Data Synchronization Network) evolved as a key data synchronization mechanism in Product Information domain. IBM Info Sphere MDM Collaborative Edition, a compelling offering from IBM for Product Information Management, leverages GDSN and provides out-of-the-box capabilities that enable trading partners to globally share trusted product data automatically. This blog is focused on GDS, inner workings of GDS, associated concepts and the GDS component in IBM MDM Collaborative Edition.

Business Scenario

One of the global water quality solution providers for residential/industrial settings had data governance issues where information was scattered across multiple regions. InfoTrellis provided them Global Data Synchronization solution to have synchronized product information across regions and retailers. This really helped business in overcoming their revenue loss which eventually led to satisfied customers.

GDS Overview

Global data synchronization (GDS) is an ongoing business process that enables continuous exchange of data between trading partners and ensures sharing of synchronized information between them at any point in time. Each organization, Supplier or a Retailer, needs to join a data pool certified and tested by GS1.

Associated Concepts:

Trading Partners – Party who is either a manufacturer or retailer of products or both are considered as trading partners.

Subscriptions – A subscription is a message that establishes a request for trade item information for a trading partner who is receiving the data on a continuous basis.

GS1 messages – GS1 is the global organization responsible for the design and implementation of global standards and solutions to improve efficiency and visibility in the supply and demand chains across sectors. The GS1 system of standards is the most widely used supply-chain standards system in the world.

Global Location Number (GLN) – A global location number (GLN) is a unique 13-digit number that is used to identify a trade location. The first 7 digits represent the company prefix. The next 5 digits represent the trade location, and the last digit is the check digit.

Global Trade Item Number (GTIN) – A global trade item number (GTIN) is a unique 14-digit number that is used to identify trade products. The first 13 digits represent the product reference number and the last digit is the check digit.

GDS Flow – GDS works based on a publish/subscribe model. The supplier is required to publish the product information to a data pool, and the data pool then matches the published data to known subscribers of the data.