TODO: (1) improve the beginning so it resonates with larger audience (not just science people)  (2) add some illustrations and examples… (3) shorten article by splitting it into several posts with links… in order to keep it higher level and allow people to go deeper when interested (when they click)….   

It’s the key ingredient of the scientific method…

This website is dedicated to it… 

No advanced technology, instrument, system, or tool can replace it.  This ingredient will help us to take maximum advantage of those tools,  but using even the most advanced technologies without it can lead to flawed results.  Moreover, low-tech or no-tech approaches that do have this key ingredient have been successful driving progress throughout human history. 

No amount of data is a substitute for it.  This ingredient is rigorously facts-based, but the analysis and interpretation of data without it, can lead to incorrect conclusions.  In famous cases of the use of the best data-driven algorithms, it has led to incorrect predictions. In some cases, it has lead to loss of life.  And when no data is available, or the amount of data is small, this ingredient can help.     

The ingredient should be a part of every scientific process, but it is not a single process, a methodology, an algorithm or an approach in and of itself. Demarcating what is ‘science’ and what it not, hasn’t been settled yet in the philosophy of science. We keep a broad perspective here and accept that there is no a “single recipe” or a simple clear-cut definition of science. That does not mean that every proposition that claims so should be accepted as scientific, we just make here our first call to examine all options and exercise some judgment.

And on the topic of what methodologies are scientific, well different problems may warrant different approaches. While some questions allow for experimentation, many questions cannot be tested in a lab or in the field. While some questions can be informed by observation and measurement, sometimes our instruments or our access reach a limit. While many systems can be modeled and outcomes calculated or estimated, these models are simplification (albeit useful) of reality and these methods may not answer ‘why’. While some theories make predictions that can be falsified (a healthy sign of scientific rigor) in practice it’s hard to know when lack or corroboration should lead to abandoning the theory vs. refining it. While probability and statistics help us gain valuable insights about problems or questions that have no clear or definitive answer, and this has proven to lead to improved decision making and outcomes, we have also witnessed many examples where flawed analytics have lead to bad and harmful decisions. We’d argue that happens when one forgets the “key ingredient” while doing “analytics”, even “advanced analytics”.

We are absolutely not anti-science. Quite the opposite, we are true believers in science, concerned about how lack of scientific or philosophical rigor can undermine trust in it, but convinced that if use the key ingredient that is much less likely to happen. Each one of the process and methods mentioned above paired with solid philosophical discipline are necessary and lead us closer to the truth.

So if the key ingredient is not a single methodology or a technology or a system, then what is it? We’d argue it is really an attitude. A way of thinking and feeling. It is the attitude that gives name to this website: “examine all”.

An attitude beyond curiosity and wonder, although the desire to know and understand is the necessary starting point.

The key ingredient is the disposition to examine everything, and to do it with an open and critical mindset. Let us unpack that in the next few paragraphs.

Let’s start with what this is not. Open does not mean undecided or lacking any belief. Critical not meaning negative or antagonistic. Everything understood in a practical way, within the feasible, not imping “analysis paralysis”, but conveying a disposition to go further in the review of our assumptions.

One can have certain beliefs even convictions based on the information available, and still consider new information or new evidence whenever it surfaces. One may or may not revise ones’ views in the light of that new information, but giving it fair consideration is the intellectually honest thing to do. Moreover, recognizing the gaps, valid arguments against, or unexplained things by one’s theory is the honest thing to do.

Critical includes questioning not only the status quo, but also questioning oneself. Top scientists think and communicate what finding would invalidate their own theory. A good analyst is not merely trying to prove that “something is true”, but trying to shoot down your own theory, and submit it to others. Recognize assumptions or biases. Consider the evidence and be willing to follow it.

So in summary…

TODO: Wrap up… edit… add posts and links with examples and more details… add references…