Data Quality Spell Book

Taxonomy, Divide and Merge (Part 2)

2009-01-29T14:19:00.002+02:00

Following on my last post, I’d like to focus on the question: What can be considered as a good taxonomy? Given the fact that taxonomy is something between art and science, a consensus will be hard to achieve. Luckily, this discussion is focused on the product data realm and based on practical aspects gained through many years of experience working with product data.

A product taxonomy should be practical

We need to have a taxonomy that enables us to search, compare, group, or analyze products quickly and easily. The pure 'academic' approach to defining categories is to group products that share exactly the same attributes, so that each group will constitute a category. This bottom-up approach will result in a long, flat list of categories. This kind of list will not serve us efficiently in searching and navigating through products and many categories will only contain a few products.

The alternative approach, the top-down approach, is based on logically dividing the product world into groups (e.g. hand, power, and machine tools as one group, fasteners as another group) and then continuing to divide those worlds into sub-worlds (e.g. fasteners is sub-divided into screws/bolts, nuts, nails, etc.) and so on. This approach results in a subjective structure and will be prone to errors. The best, practical method is a combination of the two approaches — resulting in an optimized taxonomy which incorporates categories that share the same technical attributes and includes many products as possible.

But let’s go back to the beginning. The first and most important factor in creating a taxonomy is the definition of the categories. A category should always reflect the essence or the nature of the classified object. It sounds trivial, but unfortunately, most of the categories I have come across are based on the usage of the object. For example, we had a customer (an academic research institute) that classified sugar (yes, sugar!) under several categories:

Animal food
Chemical materials
Refreshments
Office supplies

But sugar is sugar, whatever you do with it. Another common example is capacitors. A capacitor is a capacitor, but in many organizations they are divided into Electrical Capacitors and Electronic Capacitors, while the products in both categories share the same technical attributes.

(By the way, the main reason why current taxonomies are usage-oriented is because among the first to define and build classification systems were maintenance departments. For them, the best way to classify was according to the usage/facility/machine. They preferred to say "a ball bearing for X machine" than "a ball bearing with diameter D, material M, etc.”)

Notes on the relationship between categories and attributes

Another interesting aspect of categories is the ability to transform attributes into categories and vice versa. This feature enables us to optimize our taxonomy for a given organization or situation. For example, in the public taxonomy UNSPSC, there are several categories of RAM memory: Random Dynamic RAMs, Random RAMs, and Static Random RAMs. The three categories share the most of the same attributes, so if your main business is not RAMs, you may prefer to have a single category of Random Access Memory and define a technical attribute called Type with three values: Random, Dynamic, and Static.

A plate by any other name

Another factor is the name of the category. It’s important to bear in mind that we all have our own perceptions, so if category names are not simple and clear and there are ambiguities, many products will be wrongly classified. Think, for example, of the word “plate.” It may refer to coating, a dish, or board and maybe there are more meanings. It is definitely a bad category name!

In my next post, I’ll take a closer look at the category hierarchy — the second most important factor in creating a good taxonomy.

Taxonomy, The Constitution Of Product Data Quality (part 1)

2008-12-10T12:47:00.003+02:00

I think the most important, yet undervalued, factor in the product data realm is the taxonomy. The taxonomy is the skeleton and foundation of any reasonable product data quality (PDQ) strategy. A complete, well done taxonomy (and I will elaborate on this later) serves as the core domain knowledge for PDQ and increases its level of quality. Of course, having a good taxonomy cannot, by itself, guarantee high product data quality, but without it, high quality product data cannot be achieved at all – it’s simple as that.

Taxonomy can be easily compared with the constitution. A good constitution is comprehensive, clear, consistent, balanced, practical, updated, and sets limits and borders while also allowing for ad hoc judgments and decision making. A good constitution embodies and accumulates values, positions, culture, experience, common sense, and serves as a guide for the society who creates it. But having a good constitution is not enough – it should be followed, enforced, and continuously maintained. The same with a taxonomy.

To give a bit of background, taxonomy, the study of classification, is the basis for all science. We use taxonomy to structurally group similar things into categories, based on a set of common, category-specific characteristics. Aristotle made one of the earliest attempts to classify two major groups: plants and animals. Plants were separated according to size (structure)–herbs, shrubs, and trees, and animals were grouped according to where they lived–land, sea, or air.

Carolus Linnaeus (1707-1778) was a Swedish naturalist who is considered the "Father of Taxonomy." He set out to examine, describe, classify and name every living species on earth and developed the system by which we name organisms today, which groups species according to shared physical characteristics. His task was to make sense out of chaos, and to devise an organizational system that would sort out any confusion. He grouped species together into genera based upon physical similarities, and then grouped genera into families based upon broader physical similarities, etc.

In essence, taxonomy consists of four elements:

Categories (e.g. Manual Wrenches, Screw Drivers)
Category hierarchy (e.g., Assembly & Fastening -> Wrenches -> Manual Wrenches)
Attributes related to each category (e.g., Manual Wrenches: Opening Size, Overall Length, Handle Type, Jaw Material, etc.)
Values related to each category attribute (e.g., Manual Wrenches: Jaw Material: Alloy steel, Cast bronze, Aluminum-Magnesium, etc.)

In my next post, I will discuss "What constitutes a good taxonomy?"

Why Data Cleansing is Not Rational Enough

2008-11-18T16:03:00.001+02:00

Many of us use the term “data cleansing.” I have never liked this term because it actually says nothing about the state of the data before cleansing (Was it dirty? A little bit dirty?) and what is the state of the data after cleansing (Less dirty? Totally clean? Or what?) Any improvement in data quality can be considered as cleansing, though the quality remains low. What term reflects the status of the data before, after, and the value it ultimately brings?

In the product data realm, some use the term “rationalizing.” This is much better. It means that there was irrational data ("not in accordance with reason; utterly illogical." Dictionary.com) and after processing, it was rationalized ("proceeding or derived from reason or based on reasoning; agreeable to reason; reasonable." Dictionary.com). But rational is very subjective. What is rational for one person may be irrational for others. Furthermore, the term “rationalized” doesn’t even hint about the potential value.

In generic terms, what we do is take raw, crude data, run it through several processes, and produce high value data that can be considered as a "single version of truth" and as such, can be used across the organization. This process can be easily considered as refinement ("to bring to a finer state or form by purifying." Dictionary.com). Bingo! Data Refinement has it all. It embodies the initial state of the data – raw; the final state – pure; and the value – refined, pure objects are considered to have higher value.

The Beginning of Wisdom is To Call Things by Their Right Names (Chinese proverb)

2008-11-10T13:54:00.005+02:00

A few days ago, I had an interesting meeting with the CFO of a multinational company. He had recently tried to "optimize" his supply chain, or in other words, to cut costs. He knew that some products were causing him a major headache (and hole in his pocket), and had to do something about them. But which? The next step was to locate the products that account for most of the expenses (a kind of Pareto analysis) and then to find out the stock level, stock policy, average consumption, number of suppliers, the annual volume with each supplier, logistics (storage and transport costs) and so on. By doing so, he thought that he would be able to reduce the inventory and number of suppliers, to negotiate and get better purchasing conditions, and reduce the logistics costs. A good plan, indeed! Well, it is the core of Spending Data Management (SDM) and other supply chain optimization practices.

Unfortunately, in spite of the Oracle Application ERP, data warehouses, BI software, and other goodies that that the company had invested in during the last few years, he couldn't get a reliable picture of the company’s spend. I asked him to send us his product data (in a text file, Excel, or something similar), so we could analyze and evaluate the data quality.

I had a pretty good idea of what to expect, since I’ve seen it many times before. But we needed to put the evidence on the table, so to speak.

Let's take valves as a typical example. We found that they were classified under more than 20 different categories. Here are just a few examples:

Industrial Safety – Breathing Equipment – Valve/Diaphragm
Control – Control Equipment – Valve
Lifting – Winch spares - Engine/Clutch/Relay Spares
Liquid/Gas – Brass/Copper/Bronze Parts – Safety Valve
Liquid/Gas – Stainless Steel – Pneumatic Valve
Control – Control/Tubing Equipment – Electrical Valve
Control – Control/Tubing Equipment – Pneumatic Valve
Vacuum – Vacuum Installations – Right Angle Valve

In this scenario, ascertaining annual spend on valves, the valve inventory level, valve inventory turnover, and the number of valve suppliers is almost impossible. But, if all valves (irrespective of their usage) were classified under Valve, getting the required information could take a single click.

Most companies have no suitable taxonomy and, as a result, all their product data quality efforts are built on shaky foundations. If exactly the same product is classified under several different categories, decision making regarding spending and supply chain efficiency becomes guesswork.

I’ll talk more about taxonomy in future posts.

The Proof of The Pudding is In The Eating

2008-10-13T12:00:00.004+02:00

Lately, I met with a potential customer (a small-medium multinational enterprise) who (like many of our customers or potential customers) claimed there was no way the database had duplicate products. He’d established a dedicated team who was responsible for the creation of new item records (SKUs). The team has aimed to keep product descriptions as consistent as possible, inputting the product features in the same order and manner. Furthermore, the company has kept the same team for many years to ensure consistency.

Well, it’s a nice approach, but I was skeptical. I don't believe that any human being, talented as one may be, can manually maintain master data at the same quality level that can be achieved by a suitable computerized system. The proof of the pudding is in the eating, so I asked him to send us some of their data and enable our domain experts to evaluate its quality. They did. On first sight, the data looked really good, relatively speaking – the best I have seen until now. But a deeper analysis by an expert quickly revealed the problems.

Here’s a typical example. The company standardizes its product descriptions to eliminate duplications, listing each product’s diameter, steel code, length, and hardness:

Rod Dia 1" SAE4340 Lng 18' 420BH
Bar Round Dia 25.4MM SNCM8 Lng 6M 45Rc

There’s just one problem. These seemingly different product descriptions are both the same product — but using different measurements and technical standards.

· Rod is Bar Round
· Diameter of 1" is 25.4mm
· US standard SAE4340 is SNCM8 JIS standard
· Length of 18' is 6m
· Hardness 45Rc is 420BH

No one can expect that a team responsible for creating new item records can be an expert in all domains, know all the standards and common abbreviations used in each domain, be able to correctly classify and understand the various technical features relevant to each domain, or even distinguish between the varied descriptions of the same product used by different suppliers. Not to mention the huge obstacle of different languages.

The ultimate advantage of a computerized data quality system is the ability to harness and reuse the domain experts’ knowledge, creating a data quality firewall that prevents the creation of duplicate records.

Humpty Dumpty Words with Tweedledee Logic

2008-04-15T11:49:00.003+03:00

"When I use a word," Humpty Dumpty said in a rather scornful tone," it means just what I choose it to mean — neither more nor less."
"The question is," said Alice, "whether you can make words mean so many things."
"The question is," said Humpty Dumpty, "which is to be master — that's all."

"Contrariwise," continued Tweedledee, "if it was so, it might be; and if it were so, it would be; but as it isn't, it ain't. That's logic." — Alice through the Looking Glass, Louis Carroll

Many times when I review product data, I can't avoid the impression that the product descriptions were written by Humpty Dumpty following Tweedledee logic. So, “which is to be master?” That’s the challenge of product data quality.

Data Quality, Fairy Tales, Dragons and Knights

2008-03-26T12:09:00.007+02:00

"At this moment her thoughts were interrupted by a loud shouting of `Ahoy! Ahoy! Check! and a Knight dressed in crimson armour, came galloping down upon her, brandishing a great club. Just as he reached her, the horse stopped suddenly: `You're my prisoner!' the Knight cried, as he tumbled off his horse." - Alice through the Looking Glass, Louis Carroll

A recent post on the Data Governance blog mentioned this entertaining and thoughtful video about a dragon named Data Quality — indeed an innovative way to spread the data quality message. It’s a nice fairy tale with knights, dragons and all that stuff, and there’s even a moral to the story.

But in this fairy-tale, it took a costly knight with a bag full of tricks — and a really long time — before he succeeded in controlling the dragon. But it’s still a fairy tale. The problem is that in real life, there are many such knights, complete with shiny bags and glossy appearance, who all promise to conquer the dragon with almost no effort. Well, real life is no fairy tale. It takes more than a lone knight, no matter how shiny he looks, to conquer the problem. It takes powerful technology and experienced fighters to get control (and keep control) over the Data Quality dragon. No magic, no knights, no shortcuts.

To IT Or Not IT

2007-11-14T15:18:00.001+02:00

“The goal is transform data into information, and information into insight.” – Carly Fiorina

I like to compare data quality to an oil refinery. The goal of a refinery is to transform crude oil into refined, clean, and usable oil — after all, no-one will even consider using dirty oil in their cars. It’s the same with crude data: Our goal is to transform it into cleansed, rationalized, and usable information. Using crude (or dirty) data will cause a whole range of short- and long-term problems. Dirty data will prevent the organizational engine from fulfilling its energy potential, making it slow and ineffective – which presents quite a problem in the economic race against other organizations.

Back to the refinery. A refinery consists of two main entities: infrastructure and oil products. The infrastructure enables production (e.g., reactors), transport (e.g., pipes, valves and pumps) and storage (e.g., silos and containers). Oil products are the raw, crude oil and wide array of products and byproducts we all use everyday or are used by petrochemical industries. In a refinery, life is fairly “simple.” Some define what products are required, when and where; some are responsible for process design; some are responsible for the production of high quality products; and others are responsible for the infrastructure to support all these activities. Clearly, the maintenance or engineering department is not responsible for the process design, production, or marketing and distribution. Their job is to provide the required facilities for production, storage, and transportation, and to prevent contamination and reduction of product quality as a result of poorly maintained infrastructure. They are not responsible for the quality of the content.

So here’s my point: IT in a modern organization is the maintenance and engineering in a modern refinery. IT provides us with facilities to produce data (e.g., forms); they are responsible for data storage (e.g., databases); they are responsible for data transportation and delivery (e.g., interfaces, reports, queries), but they are definitely cannot be responsible for the quality of the data produced – the content.

Suppose a young engineer in a factory needs a particular screw for a production process he is working on. He will dip into the ERP system, scan many product descriptions, and a large number of products classified as screws. Among them, he finds the following four descriptions among others:

· DIN 912 10x1x30-2.9 mat304
· ALLEN SCR M10x30 stainless steel
· SOCKET BOLT M10x1 LG30 SS
· M10x1mmx30mm SHCS-SS

Well, none of them seems to be the one he is looking for, according the data in the technical catalog he is using. The next logical action will be to generate a new product in the catalog and to order this desired screw. A more expert engineer might be able see that all the above screws are actually the same, in spite of the totally different descriptions. The outcome is a new product number, new order, and inventory of a product already in stock. The next time an engineer needs this particular screw, he will conduct another search, fail to find exactly what he’s looking for, and probably generate yet another new product number with a different description. (By the way, the above product descriptions were taken from a real customer catalog!)

Can we expect IT to be responsible for the content of the product data? They provided us with the entry form; relevant fields; stored it in the database; and allowed us to restore the product data when asked. But IT cannot be responsible for the content stored in the IT systems. The main problems with data quality are with the quality of the content stored in the IT system. That is why we cannot transform data into information and information into insight.

The "install, run and ... poof" magic

2007-11-01T10:34:00.001+02:00

I spent many years on the enterprise software side, hardly aware of the existence of something called “data quality.” In fact, like many people in the ERP realm, I probably contributed to the problem, because I was focused on slotting data into the right fields without stopping to consider the actual content. Today, we’re hearing a lot more about data quality, particularly when it comes to customer data, and increasingly when it comes to product data.

It’s not really surprising that the term means different things to different people and is used for varying purposes. Data quality has become more of a marketing slogan than a well structured and defined concept, with many consultants and software companies jumping onto this amorphous bandwagon.

I’ve spent the last three years in the development of computerized systems and best practices to solve the mess we helped to generate over many years. It is tough, complex, and requires a lot of experience and know-how as well as a profound understanding taxonomy and in many technical domains.

Here’s the bad news: there are no real simple solutions to this very complex problem; furthermore, crappy data created by over the years can’t be automatically solved by a magic tool: “Install, run and… poof!”

But here’s the good news: experience and know-how, best practices and methods, suitable software tools and hard work can solve the problem and bring the quality of the data to the right level.

Lately I’ve been seeing more and more promises of magic wands and tools that automatically and painlessly fix all the data quality problems and live happily ever after. Well, I too am looking for such a magical spell book!

In the meanwhile, I thought it will be more practical to share my thoughts with those involved in the data quality realm, bring the complex issue of PDQ down to earth, and maybe save some growing pains.

I do not pretend to be objective – I am biased. I develop systems and practices, run projects all over the world and am confronted with new challenges every day. But I am going to write about the real world, without marketing hot air and without delving into the realm of theoretical concepts.

I will be happy to receive your comments and to publish your thoughts regarding the data quality domain.