HOME

Testing the model: form and function

The final stage of the systems method is to test the model to make sure that it works.

1. First and foremost, the tests must demonstrate that the model is accurate. Facts need to be checked, and assumptions need to be reviewed.
2. Secondly, the tests must conclude that the model is logically sound - that nothing important has been left out, that nothing extraneous has been included, and that what's in the model makes sense.
3. And finally, the tests must verify that the model is acceptable to the people who had the problem in the first place. A solution that nobody will use is a waste of time.

All of these are considered while the model is being developed. But errors can happen, and so a separate testing process must be applied before the model can be accepted. The tests are often done in the order presented here, though various tests can be mixed together if the designer so desires.

Testing for Accuracy

Scientific tests check to see if the data used in building the model is accurate, and if the methods used in the model are complete and correct.

For example, suppose our tomato sauce model did not include the "burner" element in its environment. If this had happened, then there would be no heat source, and without heat the mixture would not cook. Since the model shows the mixture browning, boiling and cooking, the absence of a heat source would be inaccurate.

In theory the designer checks the facts for every definition as he or she constructs the model. But in practice, this is seldom possible. Many different ideas and configurations may be tried before the designer finds one that works, and a full and complete accuracy check for each item would just take too much time.

This means that the final model may have problems, and this is why it must be carefully checked by experts who know the facts.

Testing for Logic and Common Sense

Critical tests let people provide feedback on whether or not they think the model will work properly.

There is a difference between being accurate and making sense. In the tomato sauce model, for example, it would be accurate to add the onion and herbs to the hot pan before the oil ... there is nothing in physics that would prohibit that. But doing so would cause the onion and herbs to burn, and that would be illogical.

Naturally, the designer tries not to include illogical elements in the model, but because the models can be complicated, logical flaws often appear. These can be tested by showing the model to people who have experience with the situation and asking them if the organization and "flow" of the model makes sense.

These tests also should check to be sure that the model means to others what it means to the designer. This was the main reason for making a definition for each element in the model, but nonetheless, it often happens that different people understand the model in different ways.

For example, it might be obvious to the designer that "wine" means "red wine" ... because you would never add white wine to a tomato sauce ... but someone who is not an expert cook might ask about that. Similarly, if you just dump the onions and herbs into the hot oil, they will not brown evenly, and they might burn. An expert cook might mention that the model needs to require that these items be "stirred."

The more people who review the model, and the more varied their backgrounds, the more complete this test will be. On the other hand, the people themselves may have disagreements (some people will say, for example, that white wine is fine for a tomato sauce), and more people means more disagreements. So the goal is to find a suitably large group with enough different points of view to give the model a thorough inspection.

At the same time ... the designer cannot let the model grow too large or complicated. The easiest way to resolve critical disputes is to add more detail to the model. But the reason for making the model is to simplify a complex situation in order to better understand it. In order to reach this goal, the designer is required to maintain a balance between completeness and complication.

Testing for Satisfaction

"Look and feel" tests reveal whether or not people like, and therefore might actually use, the solutions that the model will provide. It doesn't do much good to solve the problem if people are not interested in using the results.

For example, one person might tell us "I'm allergic to onions." That would mean that this person could not use our model unless we provided a substitute for onions. We would have to think about that and decide if we wanted to make the change.

As was mentioned earlier, systems models are meant to provide specific solutions to single problems; they are not meant to be general, "one size fits all" solutions. The people who have to be satisfied with the model are the people who are looking for a solution to their problem ... no other opinions are required. (The exception being legal or other considerations. A solution to "I need money" might be "rob a bank", which is fine with me, but others disagree and I am required to respect their opinion.)

The people who have the problem do not need to test the model, they need to test the solution.

At this point, one or more solutions are being proposed. If none of these work, then it's "back to the drawing board" and the entire model may have to be scrapped. On the other hand, if some or all of them are acceptable, then the designer's job is finished.

Usually, systems designers only propose solutions ... they seldom construct them. For example, I might develop a plan for making pasta sauce in an industrial setting, but I would probably not be the person who actually built the factory. I might stay on as an advisor, in which case I would be around to clarify and explain my results. If not, then only my documentation will be passed on to those who are executing the solution. It is part of my job to write a comprehensive, clear, and readable report which will serve as a kind of "user's manual" for anyone who later makes use of my model.

Doing the Tests

Testing is mostly a matter of asking questions and asking other people to ask questions. For example, given the current model for the tomato sauce problem:

Accuracy

The model might be reviewed by experts in cooking, chemistry, physics and/or other disciplines. Their goal is to make sure that our facts are correct and that we have not made any invalid assumptions.

• Are these all of the ingredients?
• Where are the amounts? How can we cook something if we don't know how much oil, how many onions, and so on?
• What temperatures should the cook use? Is the burner just turned on and left, or do we have to adjust it?
• How brown is "brown?" How do we tell when the onions and herbs are sufficiently browned?
• What steps were omitted from the process? For example, surely we stir the sauce as it cooks - why is that not mentioned?
• More ... ?

Criticism

The goal of these tests is to discover if we have left anything important out of the model. Also ... we want to make sure that we haven't put anything in to the model that doesn't absolutely have to be there.

• How long will the sauce keep? Does it need refrigeration?
• Are all the ingredients easy to find? Are any of them particularly expensive?
• What if the cooks decides to add additional ingredients ... would that invalidate the model?
• Steam removes liquid from the mixture. Is that a potential problem? Do we have a lid on the pot? Do we need one?
• There aren't any time periods associated with the model. How long should each step take?
• More ... ?

Satisfaction

The results of our work are supposed to satisfy our client. It's possible that we have developed an accurate, logical solution that for some reason is simply unacceptable. So the final test is to see how the client reacts. In this case we have produced a food product, so the satisfaction test will be based on sensory impressions.

• How does it taste? How does it smell? How does it look?
• Is it safe? What if it boils over?
• Is it easy to clean up afterwards?
• What does it cost?
• More ... ?

Repetition

If you look at my original diagram of the systems process, you will see that it contains a number of loops. I must constantly review and test the model, and based on the results of the tests, I may go back to gather more data and rebuild the model.

I do this because, as I've said many times, the process of thinking about the problem is what I'm trying to accomplish. The model is just the end product of that process.

By constantly reviewing and revising my work, I force myself to be extremely thorough. This intense focus and methodical method promotes the intersection of creativity and logic that makes systems thinking such a useful tool.

When Is the Model Finished?

The short answer is: never.

The long answer is: it's finished when whoever it has that has the problem is satisfied that the problem is or can be solved, or time runs out on the project.

Because the model is a "model", it can never contain all of the detailed reality of the actual situation. So, no matter how long you work at testing and improving it, it will never be complete.

On the other hand, we build models to help us solve problems, and the problems won't wait forever. So, at some point the designer has to decide to move beyond the modeling stage and start trying out real world solutions. This is an art, and expertise comes only with practice.

Documentation

Because these tutorials are intended for beginners, I have not spent much time on this final activity. Good documentation is essential, and it is an art in and of itself. In large projects the designer may not be the one who writes the documentation, but in any case:

• keep detailed notes of everything you do,
• revise and extend your notes at least daily as the project procedes,
• include all definitions and diagrams,
• be accurate and precise ... too much is better than too little,
• if you are responsible for the documentation, try to create it as you go along,
• if someone else will write it, try to create an outline to guide their writing.

Always keep in mind the a model may turn out to be useless without its descriptions, explanations and instructions. Because of this documentation is as important as any other part of the process.

The End of the Line?

There is much more to systems thinking and general systems theory that I have had time to present in these short tutorials. If you found what you read here interesting, you might want to take a look at the links page for more.

Examples

The following examples illustrate the contents of this tutorial:

Example 1. College Life

Example 2. Fossil Fuels

HOME