Model analysis procedure

What do you think of another model that contradicts your own model or world view? The easiest approach is to estimate the intentions of the developer of the other model, and mark it not credible or unreliable. Then you save yourself the trouble of doubting the model you most prefer. However, this is not valid if you are interested in the truth.

Why would your own model be correct and the other not? Here I will describe a method to analyse other models. It also works for models that you do agree with.

Step 1: doubt all models
Get skeptical about every statement, including your own. That means that you must become aware of every hidden assumption. Those assumptions are not necessarily false, but they could be. Also, be aware that there may be missing alternative interpretations. Get aware of every point in the model where information is attached to other information.

Step 2: find out the perspective of the model builder
What does the writer know? What does the writer assume? How does the writer get to answers? Sometimes the investigator has an approach that you have not thought of. That gives you the opportunity to learn. It is also interesting to see that a different approach sometimes give different answers, and sometimes gives the same answers. If they give the same answers, you can lower your priority for investigating the subject.

Step 3: Don’t get intimidated by experts
“It is made by Professor Noah Lott, so I suppose it is true.” Whoever made it is not important. Of course experts know more, but they could also be more biased. Even if all experts say the same, it could still be wrong. Experts often have very deep but narrow knowledge. And they also often had the same education. If a hidden assumption is in the textbooks, they might not have found out themselves.

Narrow knowledge can lead to tunnel vision, which can lead to circular reasoning. “Biology is only viewed through deep time because dating methodology proves deep time. Therefore evolution is true.” “Dating methodology suggests isochron is the most reliable dating method, because astronomy proves deep time, and isochron supports that.” and so on… Implicitly a biologist says that a dinosaur is 65 million years old, because the universe is old because of constant light speed. What does a biologist know of astronomy? If astronomers point to evolution as support for deep time, you’ll be stuck in a very big spider web of circular reasoning. Therefore the reference frame also causes bias for experts. It could be right, but it could be wrong just as well.

Step 4: Separate interpretations from pure facts
You must find out what the facts are in their purest form. “It sounds logical” is not enough. Facts are only parts that are absolutely irrefutable. Never doubt pure facts, but doubt everything else. But be careful never to consider a statement a pure fact if it isn’t. Not even if you have always thought so. Even “light speed is constant” is not a pure fact. It is a conclusion. It is either an assumption that caused circular reasoning, or a conclusion based on a lot of measurement, or a model based conclusion, or a combination. The measurements are the pure facts in this case.

Step 5: Get curious about something
If you read something interesting, ask the question: “how does the writer know this for sure?” and “how does the writer draw this conclusion?”

Step 6: Dive into the references
If the writer did not include references, then it is actually not very well written. You can search for other sources that say the same and do have references. If they all do not have references, then they are hard to check. It may be possible that it is weak, and there is no support for the statements. If there are references, dive into it until you found the most basic facts that support the statement. Usually you don’t have to read the entire references. It is quite easy to find those basic facts that support a statement.

Step 7: Check if alternative interpretations have been considered
Are there other interpretations considered? Why not? If the writer did not consider alternatives because of the framework he has, then the writer is biased and could be wrong. If the writer says “this is the most likely option, so we assume its true”, then the writer also has a chance of picking a false interpretation. The alternatives have not been tested, so that leaves open ends and uncertainties. Even if the writer says that it is so likely that he does not have to check out the alternatives: that is bad modeling.

Step 8: Find out the effect of alternative interpretations
Make different assumptions and try to find out if that gives different results. If so, then the consequences of the interpretation of the writer could be large. In some cases any alternative interpretation for a fact always results in the same conclusion. In that case, you can still doubt the interpretation, but the conclusion is true. So conclusions can be correct, even if supporting interpretations are uncertain! That is only true if all possible ways lead to the same point.

Step 9: Compare your own alternative with the original model
Can you explain things now that are mysteries to others? Or are you getting far from reality? This should influence your priority list. Eliminate impossibilities and investigate very unlikely models until you are sure they are false. That should not be hard. If you cannot prove something very unlikely is false (without it being unfalsifiable), it may be not as unlikely as you thought.

Step 10: Expose hidden assumptions
If your alternative works better, then make very clear what assumption(s) are hidden in the model you investigate. Sometimes they can be in a totally different place than what anyone expects.

Step 11: derive submodels and integrate that into your own model.
There are no 100% complete and correct models available (and there could be only 1 at most eventually), but most models are not entirely worthless either. Remember, you do not do this to make yourself feel better or proud, to win or to discredit the other model or its investigator. You are searching for the truth. Therefore, you can basically detach every part of the model you consider uncertain or incorrect, but in most cases large parts remain in tact. Do not throw that away! Even if it is from somebody with a world view that you disagree with or somebody you don’t like at all. It’s valuable. The parts that remain in tact should be integrated in your own model. Get to step 12 in the model building procedure and continue. 

<- Previous page: Model development procedure     Next page: Definitions ->

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s