ePortfolio: Dr. Simon Priest



THE RESEARCH INQUIRY PYRAMID: The Research Pyramid (1994) explains my philosophy regarding inquiry. It describes a sequence of steps that I believe researchers ought to take when investigating a phenomenon (called "IT" here). Answering a good research question leads to more good research questions. Therefore, I always recommend that future inquiry direction should be built upon past evidence-based inquiries. Otherwise, researchers would take a scattered (shotgun) approach to investigation and much time, money, and other resources would get wasted.

I like to use the example of earthquakes or cancer to explain this concept. If we attempted to predict and control these (causality and discrimination points where studies are currently focused), without understanding influence, relationships, differentiation, or description, then we would most certainly fail. However, by initially reporting what these phenomena look like, how they are similar to other geothermal events (volcanic eruptions) or other diseases (failed immune systems), what else is associated with them, and what can impact them, then we build a strong foundation to inform our inquiry toward stabilizing cities or finding a common cure.



TRANSDISCIPLINARY INQUIRY: Students often confuse multidisciplinary, interdisciplinary and transdisciplinary terms, so I made this graphic in 1995 to help them see the evolving difference. Late one night, I was explaining to a doctoral student my thoughts on the difference between basic and applied inquiry with respect to interdisciplinary and transdisciplinary inquiry of complexity with diversity. I drew this diagram of four purposes for inquiry on a napkin; here is a better version from that same year.



POSITIVISTIC vs. NATURALISTIC PARADIGMS: I was raised a hard scientist with my university study of biology, chemistry, physics and mathematics. By 1987, I was a devout positivist (deductive and objective scientist) and my then partner was a confirmed naturalist (inductive and subjective artist). Her 1988 dissertation conducted three studies: a statistical analysis of quantitative data collected under positivism, a thick and rich description of qualitative data on the same topic conducted under naturalism, and a comparison of the knowledge generated by both approaches seeking "truth" from evidence. She won a top award for her ground breaking research. Together, we developed this comparative table by debating philosophy around the dinner table and on long dog walks. In the years that followed, I studied human behavior and morphed into a Pragmatist (between these two extremes).



PRAGMATIC PHILOSOPHY AND INQUIRY: The Latin and Ancient Greek terms of “pragmaticus” and “pragmatikos” mean “fit for taking action or doing a practical deed.”  Pragmatism is about taking action for the purposes of bringing change.  It encompasses a philosophical movement, a set of ethics, and several theories about truth and knowledge.  Although not absolute, I am mostly a Pragmatist in both philosophy and inquiry.

Philosophy: Pragmatists believe that the universe is constantly changing and so a phenomenon cannot be examined in isolation; it must be studied, confirmed, predicted, and tested within the human experience.  To this end, they emphasize the practical application of ideas through action and active inquiry.  This dynamic philosophy allows pragmatists to confidently hold a combined approach to their research.

Inquiry: Pragmatists merge the best of deductive approaches from positivistic inquirers with the best of inductive approaches from naturalistic ones.  When deduction or induction is not possible, pragmatists use abductive inference to reach the best explanation.  Pragmatists seek to explore emerging ideas and confirm established thoughts.  They do not see the world as static and so they cannot concentrate on one small isolated piece of a phenomenon.  They prefer multi-level studies, where the interaction of all pieces is critically considered.  Their ontology (the nature of reality) is a mix of objective and subjective perspectives.  Their axiology (the nature of perceived values) is a blend of value-free and biased-rich interpretations.  Their methodologies bridge qualitative and quantitative systems based on the research questions they ask and are not locked into either set of techniques.  They are like photographers who choose their lens based on the image they wish to create.  With regard to epistemology (nature of knowledge) pragmatic researchers consider five sources of information to be equally legitimate (E L I T E):

  1. Empiric (objective facts established by examination, demonstration, and systematic inquiry),
  2. Logic (new explicit information created through conscious analytical and creative reasoning),
  3. Intuitive (subconscious data, based in uncertain feelings, but without conscious reasoning),
  4. Tacit (inherent, implicit knowledge that is not easily transferred from one person to another, but can be gained experientially by personal or skillful observation, imitation, and practice), and
  5. Expert (information from authentic authorities as found in refereed articles or presentations).




BRIDGING QUALITATIVE / QUANTITATIVE GAPS: I grew up in Greater Vancouver and this photo of the Lions Gate Bridge over the First Narrows is a perfect metaphor of bridging the gap between quantitative and qualitative data. Regardless of one's affirmed paradigm for conducting research (Naturalistic or Positivistic), the bridging solution remainsthe same.

Researchers are like photographers. The Positivist photographs wildlife and frequently chooses a telephoto lens to get in close to their subjects. The Naturalist photographs scenery and often selects a wide angle lens to take in the whole view. Since the photographers seek to create different images, they pick different lenses. As researchers, they ask different questions and therefore gravitate toward using different methods. The Positivist prefers to measure quantity, while the Naturalist prefers to describe quality.

However, sometimes the wildlife photographer will use the other (wide angle) lens to gain a broader perspective of the entire herd and ocassionally the scenery photographer will switch to the other (telephoto) lens to capture a particular object of interest. Researchers are the same. Despite their predetermined research philosophies and inquiry paradigmal preferences, many are deciding to utilize both types of data in the same singular study.

The content added within this attractive picture lists the key hallmarks of each data type: its name, definition, formats, sources, examples, and analysis techniques. The key factors that may bridge the gap between them are listed in the middle. MEDIUM sample size is the first opportunity. Since large randomized samples are more common in quantitative and small purposeful samples fit better with qualitative, the middle ground of medium size enables the use of both data analyses. MIXED methods can be sequential (one before the other or in reversed order) or convergent (both in parallel, then joining together). Since one approach tests hypotheses and the other creates theory, doing BOTH can be a common focal point for the same study. Data can be converted from one type to the other. Triangulation involves using other data or one data set to confirm and corroborate another. Longitudinal designs offer the chance to combine methods as both can examine changes over the long term. Irrespective of beliefs, researchers can partially integrate these two methodologies by many means.



ABORIGINAL AND INDIGENOUS METHODOLOGIES: I am painfully aware of the irony (and perhaps the total hypocrisy) of studying Indigenous/Aboriginal (I/A) populations using I/A methods, while not being aboriginal or indigenous myself.  In fact, from time spent living with and studying dozens of I/A groups, I know that many were initially skeptical of my presence and so I would never mention the term research.

“The word itself, ‘research,’ is probably one of the dirtiest words in the indigenous world’s vocabulary….  It stirs up silence, it conjures up bad memories, [and] it raises a smile that is knowing and distrustful.  The ways in which scientific research is implicated in the worst excesses of colonialism remains a powerful remembered history for many of the world’s colonized peoples…[one] that still offends the deepest sense of our humanity.” – Linda Tuhiwai Smith (1999, Decolonizing Methodologies: Research and Indigenous Peoples, Dunedin: University of Otago Press, p 1).

I spoke with the above author about the probable irony and my possible hypocrisy, when I was a visiting scholar in New Zealand.  She and other I/A researchers (Chilisa and Kovach) have suggested that Euro-Western researchers (like me) should open up their preferred methodologies to make new space for I/A methods.  After trying these out, Euro-Western researchers should evaluate and modify as appropriate.  All three of these I/A authors emphasized the importance of context and culture.  We must challenge and change the base suppositions, core inspirations, and foundative principles that enlighten our research practices.

So I started studying the folks I lived with for program evaluation purposes.  In the beginning, I spent a great deal of time building trust and relationships in a friendly way.  Early on, I developed a penchant for qualitative methodologies, which were popular in the anthropological literature long ago.  My partner, who accompanied me in those days, was a devout naturalistic inquirer and she often led the way.  However, I can’t help but recall from those experiences that some of our approaches were better suited than others and fit only with certain cultures.  For example, in those cultures where story telling was revered as oral tradition for passing down history from Elders, we wrote down the stories that were told.  However, when this was not cultural tradition, we couldn’t find anyone who was willing to tell a story!

I began by relying on conversation, interview, and observation.  These three were the mainstay of qualitative methods at the time.  Initially, my perceptions were heavily colored by my culture and I knew I was being very biased in my interpretations of stories, answers, and actions.  I started to worry that my perspectives might marginalize the I/A subjects.  However, as I grew steadily into their cultures, my perceptions changed and my interpretations followed.  The so-called “going native” phenomenon, which may be frowned on by some, is a very useful and necessary occurrence to reach an uncolonized analysis.

So I quickly came to realize that I/A research using I/A methods is simply good research with sound methods applied to, for, about, from, and with I/A peoples.  So I began training members of the I/A groups to be secondary researchers and interpreters for small projects.  They provided the integration with culture that I could never have obtained no matter how long I lived with each group.  I was born outside and had developed outsider assumptions, beliefs, and ideals that would remain with me for life.  As we worked together on research projects, these I/A researchers shared some unique methods.

  • Talking Circles or group discussions where the chief’s staff is passed around as a talking stick and possession allows only the holder to speak with the expectation of silence from all others (common to several cultures in Western Canada and Africa)
  • Journeying or the generation of knowledge of nature and culture from shared experience while travelling on the land and water (Ngati Purou Maori from the East Cape of New Zealand)
  • Singing and Yarning or the narrative analysis of oral history from the Dream Time of creation to present day through songs and stories (Koori and many others Aboriginals in Eastern Australia)
  • Illustrative Iconography or the traditional ceremony-based interpretation of ritual art such as pottery symbols, kiva murals, and the Matachina dance (“The Painters” – Picuris Puebloans from the southwestern USA)

Interestingly enough, my experience with all of this was in English (the dominant language for academic research) and so all content was filtered through a secondary translator or perceived and expressed by the primary source (for whom English was not a first language).  Either way, the analysis of meaning and message was wide open to my misinterpretation.  This was the greatest challenge for me as a qualitative researcher: not interpreting through my own culture, but understanding enough about living in their culture to do justice to their words and actions.  It seemed to me that any I/A research ought to benefit the culture in context.  At that time, I thought of 13 elements that were important, but first, a story.

One day, when I was out walking on the open savannah with an African friend, I found a sun-baked turtle shell with 13 plates on the outside.  It was missing the lower plastron (belly plate), so I could see inside the shell.  The spine was arranged along the inside center line and the hips were clearly visible.  Knowing my basic anatomy, it was obvious to me that the 13 plates on the outside shell were vestigial ribs and shoulder blades that had evolved into plates.  My friend had a different interpretation and told me that the first turtle was created by the gods with a smooth shell, but when they delivered it to earth, they accidentally dropped it on its back and cracked the shell into 13 pieces: one for each of the 13 lunar cycles in a year.  Both interpretations made sense to us.  Mine was founded on evolution and anatomy, while his was based in a seasonal life, lived by the cycles of the moon.  We fully understood one another. 

Our differing perceptions of that turtle shell provided me with a nice metaphor for how context can make all the difference in interpretive data analysis.  I put the shell back where I found it, but he later returned to pick it up.  When I departed the tribal village, he gave it to me as a gift and I thought using it to underlie the platform for these 13 contextual concepts of a cultural research schema would be fitting (see diagram below). 

The center five plates are concerned with the overarching I/A geopolitical philosophy that will govern culture, research, and the interactions of both.  The right four plates are about culture within research or the historical elements that will influence study.  The left four plates are about research within culture or the inquiry elements that will be traditionally determined.  All plates concern research where at least one of these is I/A: the researchers, the questions asked, or the study location and its members.

  1. Community Empowerment and Autonomy: This is the principal purpose of I/A research and so is placed at the center of the turtle shell.  In Euro-Westernized research, our imperial methods and reasons have unwittingly disempowered I/A groups and diminished their identities.  Studies conducted with I/A subjects must lead to self-determination and betterment of community.
  2. Interest and Need: Consent to engage in I/A research will depend upon the interests and/or needs of the community as guided by their dominant paradigm of ontology, epistemology, and axiology (see below).  Perhaps they share a need to understand local phenomena or create I/A theory.   Maybe they are interested in sharing their experientially derived knowledge regarding research or integrating their paradigm or methods with Euro-Western approaches.  To begin and ensure future success, be certain that interests and needs are aligned for all parties.
  3. Collaborative Agreements: The research must be planned through a process of equal and collegial collaboration (not just cooperation).  The I/A culture may value community consensus or Elder pronouncement, so be prepared for and work within either agreement seeking process.
  4. Permissions and Protocols: Protocols refer to behavioral rules within a culture.  Researchers will need to know these and understand where research might interfere with traditions.  Permission must be given for all aspects of the study from early conception, through data collection methods, to the terms of how data or findings are privately handled or publicly shared.
  5. Council Consultation: The governing political structure, usually a council or group of Elders, will need to be consulted and sign off on all of the above, once each has been consensually decided.
  6. Healing and Recovery: at the very least the I/A culture will have suffered hurt at the hands of historical colonization like loss of ancestral territories, forced relocation to reservations, or removal of children to colony schooling.  Omnipresent anger from pain may limit not only the products of the research, but also its processes such as trust, access, and willingness to share.
  7. Truth and Reconciliation: at the very most, the federal government (of South Africa, Australia, New Zealand, Canada, but not the USA) will have struck a commission, undertaken a democratic process, and/or honored a signed treaty, to address and rectify the historical mistreatment of I/A populations.  The outcomes of this commission, process and/or treaty will recommend or require certain constraints that may restrict the research, but these must be obliged and pursued.
  8. Owning and Managing: In the past, information that was generated by research left the culture.  The researchers benefitted, but the culture did not.  Often, the culture was lessened by the loss of its contributions and felt short changed by researchers not giving back in return for sharing.  Today’s I/A groups should be given ownership and control of all relevant data.
  9. Sharing Findings: Results should be presented in language that the culture can understand.  This may include graphics, maps, and photographs rather than words.  The I/A community gets to decide how and where to share findings, so seek authorization to publish well in advance of conducting the study.  Some I/A individuals may not see the worth of sharing outside of the community.  They may prefer to serve community needs first and go no further with the dissemination of findings.
  10. Ontology or the Nature of Reality: The I/A paradigm or guiding philosophy is made up from the ontology, epistemology, axiology, and methodology of the culture.  The nature of reality for I/A groups is often about constructing multiple and subjective realities that do not exist until the researchers perceive those realities.  A commonly accepted reality can be forged through open community discussion, because all members have knowledge and their interpretation is valued.
  11. Epistemology or the Nature of Knowledge: Many I/A paradigms share some beliefs about knowledge being personal, natural, heartfelt, researcher entwined, and experientially based in practice.  Knowledge is generationally passed down through oral tradition and ritual ceremony. 
  12. Axiology or Values and Ethics: Respect and reciprocity in relationships seems to be at the heart of most I/A paradigms.  Genuinely giving people, they simply don’t want to be taken advantage of, especially given past histories.  Understand the values that underwrite all behaviors in an I/A culture.  Be sure to avoid transgressing these while negotiating and researching in that culture.  Build genuine and trusting relationships with everyone connected to the research study.
  13. Methodology or Procedures and Frameworks: In the typical I/A paradigm this tends to be more studying within bounding frameworks than specifying unique I/A procedures (see the examples listed above).  The framework is created from the researchers’ interpretations of the guiding I/A philosophy or cultural paradigm noted here.  So no methods are suggested; researchers choose their questions and these determine methods, and so on.  Everything takes place with the researchers frameworks extracted from their own experiences and interpretations.



COMMUNITY-BASED PARTICIPATORY RESEARCH: This is a natural corollary of the Indigenous and Aboriginal Methodologies (above).  Community-Based Participatory Research (CBPR) is a sustainable approach where internal community members and external researchers partner as full and equal colleagues to conduct a study.  Internal stakeholders are engaged in all stages of the study from initially asking the research questions, through collecting the data, to disseminating the final results.  All partners contribute to making decisions, sharing expertise or knowledge, and owning the outcomes.

CBPR has its roots in Health and Social Work.  As such, its principal aim has been to improve the society’s wellness and quality of life.  However, one subtle difference from a participant-dominated study is that community stakeholders are also partners in determining the recommendations and application of the research.  They determine the policy arising from a study and its execution as social transformation.

One of my earliest uses of CBPR was a healthy community initiative concerning water.  My partner, was directing this change effort and asked for help with the research component.  We gathered 100 key stakeholders to: formulate a research plan, identify and measure the impacts of water quality on health or economy, and collect information to establish the current inventory regarding all aspects of water from septic and fertilizer runoff, through watershed degradation, to saltwater impacts on salmon and oysters.  One outcome of the study, a materials quality report card called “Water: a precious resource" was published in 2004. It resulted in regular water quality monitoring by Mason County, a increase of citizen awareness, and community-based actions to protect water quality.

This CBPR approach was replicated with some of her other Mason County healthy community initiatives such as: domestic violence, teen pregnancy, housing needs, education, and rural wellness issues.  For many of these concerns, the process became iterative in four cyclic stages: study issues, share outcomes, take action for change, evaluate progress, and then repeat the action research cycle.  I drew this map to trace our CBPR steps in a partnership atmosphere of constant maintenance and sustenance for all parties to the study.



INDIGENOUS COMMUNITY KNOWLEDGE: These ideas stem from a workshop I did around 2012 for American educators.  At that time, the word “native” was their accepted parlance.  To internationalize this content, feel free to substitute your country’s preferred term in current usage such as tribal (Africa), Maori/Pacific Islander (New Zealand), aboriginal (Australia), autochthonous or original (others).

In this instance, community is defined as a social unit bound together by conventional wisdom that includes spiritual beliefs, normative behaviours, cultural identity, and/or traditional values.  These mutual attractants bring community members to collect in a common place, so they can live, co-exist, interact, and share resources in a unified society.  This societal unity strengthens their social ties, fulfills their needs, and perpetuates their ancestral relations, thereby reinforcing a solid sense of community.

Knowledge is the theoretical and practical understanding of something or someone.  In order to qualify as knowledge of these subjects, the tripartite criteria of a “justified true belief” must be fully satisfied.  First, people must believe in their thoughts; justified truth that is not believed cannot be knowledge.  Second, their thoughts must be based in accurately factual information; falsehoods do not qualify as knowledge.  Third, people must be able to justify the reasons behind their comprehension and this is usually accomplished by inferentially associating new evidence with old understanding.  However, this tripartite approach is imperfect and many common exceptions are possible, such as lucky guesses (lacks belief), false assumptions (lacks truth), and learning based on unreasoned information or simply wishing it was true (lacks justification).  Nevertheless, knowledge can be organized into a family tree of several types with these definitions.  Please note that disagreement among philosophers, repeatedly uttered during debates that have lasted for centuries, makes these definitions flexible and more like opinions.

  • EXPLICIT = openly communicated as conscious science, taught and learned via education systems, can be easily expressed, remembered, explained, and conveyed to others (stored in the mind or on media)
      • Propositional = describes and declares statements of factual information gained via reasoning
      • Non-propositional = how one feels, thinks, behaves, and believes as well as information learned
  • IMPLICIT = tacit, hidden from conscience, private not public, traditionally discussed within a community, difficult to verbalize, write, articulate, and transmit to others (stored in the mind, but not on any media)
      • Technical = skills to act or perform that are innate or derived from a unique set of experiences
      • Cognitive = ingrained perception, awareness, and interpretation of data that shape our worlds
  • A priori = non-empirical information that is independent of previous experience , but requires reasoning
  • A posteriori = empirical information learned through experience and reasoned reflection on experience
  • Acquaintance = information about subjects from sustained interactions with those people or things
  • Procedural = how to correctly/best perform a task or do something, using gathered and practiced skills
  • Conceptual = information about thoughts, ideas, and impressions regarding laws, theories, and models
  • Common = pervasive and valued information that is widely accepted by a community without question
  • Self/Personal = intuitive information about one’s self that is unknown to any other community member
  • Cultural = information embedded in a community’s traditions, art, songs, stories, dance, and subsistence
  • Ecological = information about the local nature, ecosystems, ekistics, and sustainability of a community
  • Spiritual = information concerning religious deities and/or sacred beliefs treasured by the community

I have represented these connections in a “mind map” that is shaped after Anishinaabe-style artwork (with permission).  In Anishinaabe tradition, as in many other cultures around the world, the owl is a symbol of wisdom.  Its feathers are a gift from the creator (Kitchi-Manitou) to protect against harmful spirits and attract helpful forces during a vision quest (meditation separated from community in a search for the meaning of life as inextricably connected to creation).  The outstretched wings of the owl divide and disperse the many kinds of knowledge in their collective family tree. Emanating from the wings are several “streamers” (known by Anishinaabe to be flowing lines of communication and interdependence amongst all living things) that terminate at several examples of indigenous knowledge.

Note that the left side contains five examples that are singular, different, and isolated from one another, while the right side integrates the five types of knowledge unique to indigenous communities into one example.  Despite its Euro-westernized theory, the knowledge examples on the left can be codified and spread throughout the community with a high degree of transference.  On the right, weather prediction draws on information that every community member knows about prevailing winds (common), changes of feelings internal to one’s body (self/personal), the sense of seasonality (cultural), plant or animal behaviours, where birds nest and hide or tree branches droop or rise (ecological), and a strong kinship or affinity for land, water, and air (spiritual).  Only in this combination is weather prediction possible.

For the inhabitants of Euro-western settlements, the competence to predict weather has been lost from living an insulated life indoors and relying on the Internet, while others with different perspectives will have learned to forecast from living an exposed life outdoors with a necessity for survival.  And this is how indigenous community knowledge is unique.  It blends information obtained through close and personal interaction with the natural environment coupled with the desire to survive, but also to live sustainably.  Understanding is based on continuing observation and analysis adapted to suit the local circumstances.  Knowledge is passed on to future generations through song, dance, and storytelling.

Indigenous community knowledge merges content with format and process with product.   One cannot tell where Anishinaabe legends begin or end, if a sweat lodge is spiritual or medicinal, whether certain plants are meant for sustenance or sacred healthcare, and when a pow-wow is given as entertainment, expression, or education.  And this doesn’t really matter, because indigenous community knowledge is about “and/both” rather than “either/or” with a healthy dose of amalgamation blended in.  In an effort to protect diminishing culture, language and oral history, some of this knowledge has to become taboo.



INDIGENOUS DEVELOPMENT AND RESEARCH: Under the umbrella term of research and development (R&D), indigenous development is a very different matter than the commercialization of a new product or service discovered through scientific inquiry.  In fact, the term “development” may not be the best word to use.  What has “development” brought to original cultures?  An examination of Euro-western influences on indigenous people such as the Tswana and Kalahari Bush”men” of Africa, the Aboriginal Koori in Australia, the Polynesian Maori in New Zealand, the Inca and Amazonia of South America, the Aztec and Tarahumaran of Central America, and most First Nations in North America, have all shown the same pattern.

  1. Initially, the epidemic diseases of the times were introduced: smallpox (used purposefully as a bio-weapon by the American military in 1763), bubonic plague, cholera, diphtheria, measles, typhoid, tuberculosis, pertussis, influenza, and HIV/AIDS. This led to staggering numbers of dead original inhabitants.
  2. Those remaining were prevented from subsistence living as a community; they were no longer allowed to hunt, fish or farm by traditional methods. 
  3. These communities were relocated onto “reserved” lands, usually with lower quality resources. 
  4. Some community members were enslaved to extract high quality resources from nearby lands.   
  5. Pollution from resource extraction contaminated culture and sustenance sources leading to diminished knowledge being passed on to future generations for fear of poisoning. 
  6. Each generation’s children were sent to religious schools, where they were unfortunately beaten or abused in other ways. 
  7. Many children refused to attend and chose to run away, only to be further punished when later caught. 
  8. Community members were treated as second/third class citizens, where most were badly handled and many by extremely inhumane methods. 
  9. Within community members, compensatory behaviors arose such as fighting, stealing, and other crimes or ways of acting out. 
  10. Drug/alcohol abuse became an accepted way of escape for some members; suicide developed into a prevalent final solution for others.

How can this be called development?  How can we stop repeating this cycle?  To begin, we need to substitute a more appropriate word for "development."  In the past, I have used "learning and growth" (for corporate training and development), "enhancement" (for staff or faculty professional development), and "transformation" (for student learning development).  In terms of indigenous populations, we should let them choose a term from their own language that should ensure their cultural unity and independence from Euro-western ideals.

Next, we need to change the way we conceive of development. It should not involve forcing a predetermined plan on the community (despite having fully researched such a plan). Instead, it should engage community members in a collaborative process that ensures self-determination with the autonomy to decide on what is wanted and where research can enable this collaboration (see the community-based participatory research model above).  The role of Euro-western “experts” should be to give advice when asked, to suggest ideas when something might be missed, and to conduct research that provides evidence to gauge success or failure and to make future decisions. 

Since truth can be created and perpetuated by the repetition of an obvious lie (just ask Mr. Trump), a public and transparent examination to expose the exactness of what was perpetrated in the past is also very necessary.  Canada, Australia, New Zealand, and South Africa have all had versions of their own “truth and reconciliation” hearings.  America hasn’t yet had the important national conversation needed to consider taking this initial step.

In Canada, the 2015 Truth and Reconciliation Commission’s “Calls to Action” finally addressed issues of child welfare, education, language and culture, health, and justice.  These also sought to settle other reconciliation matters regarding established rights, political relations, jurisprudent equity, political process and policy, civil servant training, religious apologies, school curriculum, youth programming, museum archives, missing people, commemoration, media, sport, business, and immigration. The federal government is making measured advances in most of these concerns.

In an effort to enact the "Calls to Action," local elementary, secondary, and tertiary education institutions across the nation have been “indigenizing” their staff, faculty, students, facilities, research, curriculum, instruction, and other elements of inquiry, teaching and learning.  This is a particularly important step, given the past history of Euro-western governed education and indigenous populations.  By inappropriately schooling its youth, solely in non-indigenous curricula, any culture can be completely erased over one generation of time!

The last step requires equal partnerships among all parties to ensure the enlargement of indigenous leadership influence and the reciprocal sharing of knowledge for the sustainable benefit of all.  Indigenous learners should be especially supported (where needed with unique pedagogies) to conduct research and rise to positions of authority, where they can legitimately make a big difference in each small community. Finally, without betraying sacred content, indigenous community knowledge should be shared, explained, and analyzed through scientific methods, just as Euro-western knowledge should be shared, explained, and analyzed within indigenous culture.



THE TWELVE V’S OF BIG DATA ANALYSIS: I started working with big data in 2008 and extrapolated some extra V-words in 2010 to describe successful characteristics of big data management.  Consider these explanations, where data is the plural form of the singular datum.

With ongoing improvements in data capture and storage, and the advent of the “Internet of Things” (where most devices and appliances are networked), enormous amounts of information are being generated.  A size limit (ex: 1 Terabyte or larger) does not determine the classification of big data sets.  Instead, big data are defined as any information set that cannot be analyzed by traditional methods or statistics.  Big data require a uniquely pragmatic approach to their gathering, secure storage, and analysis.

The purpose of analyzing big data is to make predictions, often about human behaviors.  For example, epidemiologists use big data analysis to spot patterns in the spread of disease.  Economists use this to identify business or market trends.  Meteorologist and climatologists use it to forecast weather and track changes in global temperatures.  Each of us benefits from big data analysis when we use a map app to plan our route to a destination.  Traffic cameras and the movement of every GPS-enabled cell or mobile phone is recorded and analyzed to determine traffic patterns (slowdowns and gridlocks) that are then fed back to mapping applications on those same phones, so we can choose the best route to follow in rush hour.

In educational administration, I’ve spearheaded big data initiatives for my institutions at four levels of analytics: tracking uncompleted assignments, poor academic performance, and lack of contribution to team tasks (learning/teaching level), real-time aggregate presentation of student enrollment, retention, probation, and expulsion (academic/administrative level), effective sources of marketing information correlated to new registrations (executive/institutional level), and measuring e-learning quality and adaptive learning success to influence policy and accreditation standards (government/legislative level).  Nevertheless, higher education has fallen well behind other sectors in big data analytics and applications, due to an extreme shortage of qualified staff.

Analysis of big data can avoid sampling and simply consider the full data set.  Alternatively, a set can be sampled.  The former, analyzing the entire population, does not need to adhere to the usual statistical assumptions of parametric procedures (used to estimate the sample’s representation of the whole population).  While the latter, sampling some of the population, definitely does require statistics and comparison with the normal curve remains compulsory. 

Four V’s have been coined to describe the characteristics of big data: volume, velocity, variety, and veracity.  To these I added eight more to expland the considerations of managing big data: virality, value, viscosity, variability, validity, verifiability, viability, and visual.  This wheel diagram summarizes the twelve V’s of big data.  

  1. Volume (quantity): can you find what you are searching for in the generated sea of data?
  2. Velocity (speed): are data captured and processed rapidly enough to meet decision demands?
  3. Variety (type/nature): do the collected data portray the breadth and depth of the construct?
  4. Veracity (quality): are the data informative or disinformative and do they impact accuracy?
  5. Virality (dispersion): can outcomes be rapidly circulated and decisions placed into action?
  6. Value (usefulness): are the data and their analysis/predictions helpful for making decisions?
  7. Viscosity (resistance): is data flow unrestricted or bottlenecked in the analysis processes?
  8. Variability (inconsistencies): are errors present and do these hinder the overall analysis?
  9. Validity (precision): do the data measure the construct and variables they purport to measure?
  10. Verifiability (reliability): are the data consistent representations and truthful measures of reality?
  11. Viability (sustainability): Is the data gathering going to last or will predictability remain feasible?
  12. Visual (image/idea): are the analyses outcomes or predictions easy to understand and present?



ELEMENTS OF E-RESEARCH AT THE EXTREMES: Electronic Research or eResearch means conducting evidence based inquiry (research) in an online (electronic) setting through reliance on information and communication technologies (ITC).  While eResearch can be as simple as conducting a survey online, I am excited by eResearch at the extremes, where collaborative thinking, cloud or fog-based grid-computing, and big data sets are engaged.  Since I proselytize that the current solutions to the world’s great problems (water shortages, food imbalances, climate change, peace, poverty, etc.) will come from crowd-collaboration through technology, I am keen to conduct eResearch and support faculty in their efforts to do so, as a natural fit with this necessity and eventuality. 

The key difference between current or future eResearch and present or past traditional inquiry is the use of distributed locations.  The eResearchers are found in varying geographic places.  Big data are acquired from dispersed sources and stored in the cloud (spread across many server locations).  The analysis takes place via grid-computing networked across scattered sites.  Collaborative discussion takes place over ITC.  Findings are published and disseminated through the Internet (email), on websites, in electronic journals , and via web conferences.  Traditional inquiry saw researchers working in isolation or small teams, keeping data on their own servers, conducting analysis on their own computers, not collaborating openly, and disseminating findings via print journals and in person conference attendance.

Successfully carrying out eResearch at the extremes requires the inquirer to consider nine interactive and important elements.   Based on my past efforts in eResearch, these nine are organized into three groups of three (research conduct, cyberinfrastructure, and other resources) as shown below.



TECHNOLOGY TRANSFER AND COMMERCIALIZATION: In my academic administration and business leadership roles, I’ve played various parts of the technology transfer and commercialization (TTC) sequence from researcher and evaluator, through protector and investor, to marketer and developer.  In working with innovative software or training, invented facilitation techniques, improved models or theories, and imaginative fashion with wearable technology, I created a spiral to explain the TTC sequence.

Technology transfer involves the dissemination of new knowledge or information, tools or machines, and methods or processes from the original point of innovation, invention, or improvement to people, places, or other parties.  For example, a new way to test for a disease, discovered by researchers using university resources, needs to be further developed within a joint venture that fairly includes the inquirers, their institution, and other funding sources. Commercialization involves distributing this transferred content to a sales market where profitability is possible.  Continuing the above example, once fully developed, the disease test is mass produced for consumption by hospitals and medical labs.  Eventually, a home version may be created and sold “over the counter” to the average customer. Commercialization sometimes follows technology transfer, but these can also occur independently of one another.  The spiral sequence describes the two together.

  1. Discovery: invention, improvement, innovation, idea
  2. Disclosure: intellectual property (IP), full description,  specific detailed criteria
  3. Evaluation: viability, feasibility, expert feedback (distinct/sustainable competitive advantage)
  4. Protection: IP ownership, copyright, patent, (with clear liability/accountability)
  5. Formation: licensed startup company, with optional franchises, and corporate partnerships
  6. Marketing: commercial analysis, market research, business plan (unique elements of distinction)
  7. Development: build and test prototype, scalability, ongoing refinement, production
  8. Revenue: distribution, sales, profit sharing, reinvestment, benefits to society, growth

In the research phase, discoveries are made and intellectual property (IP) is disclosed to the sponsoring institutions.  In the safeguard phase, those institutions work with the researchers to evaluate the likelihood of success and move to protect their IP.  In the organize phase, a company is formed to take the IP further,  market the IP, and develop strategic plans for commercializing the IP.  In the commercialize phase, the IP is further developed and sold with revenues being shared and reinvested in additional research as the TTC sequence repeats.



FIVE KINDS OF PROGRAM EVALUATION: Most of my students get confused about program evaluation and mix everything together into a single amalgam. In order to understand the five kinds of program evaluation, one must first understand the five phases of program development, because evaluation is conducted at different places in this developmental sequence. Jude Hirsch and I created these phases (2002), building on earlier work I did with Lee Gillis and Mike Gass.

  • DIAGNOSE: Assess need by multiple methods/sources (use combination of conversations, surveys, interviews & observations)
  • DESIGN: Plan program based on assessed needs/desires (refine purpose, goals & plan LETS: logistics, events, timing, staff)
  • DELIVER: Present program as planned, but be flexible (include intro, body of events, conclusion & remain open to changes)
  • DEBRIEF: Facilitated discussion around events (facilitate by funneling, front-loading, framing, freezing, focusing & fortifying)
  • DEPARTURE: Closing activities before clients leave (action planning, evaluations, anchors/souvenirs & schedule follow ups)



COMPARISON OF THE FIVE KINDS: As you can see in the table below from the year before, each type of program evaluation (and its "also known as" names) fits into the sequence at a different location and for a different reason (2001). The methods used are the same as in social science research with descriptive data analysis. This isn't rocket science, but you do need a map to avoid getting lost.



A PLAGIARISM FLOWCHART: Plagiarism can be defined as proposing the ideas of another as your own or presenting your work as original and accurate, when it was actually constructed from existing or falsified sources.  Plagiarism indicates a lack of academic integrity and is closely related to similar forms of academic dishonesty such as: cheating, fabricating information, bribing, sabotaging, and misconduct.

Over the decades, I have experienced some plagiarism of my academic work ranging from well meaning practitioners copying single publications, through competitors duplicating entire websites, to trusted employees presenting my research as their own at international meetings and claiming to be my chosen representative.  In their defense, their explanations were always the same: “imitation is the sincerest form of flattery!”   Perhaps, but none of these examples included any form of requested permission or correct attribution.  And herein resides the heart of the plagiarism issue.  If they had asked for prior permission and given proper recognition, I would have said “yes, no problem, just fully cite my work.”

We must give credit where credit is due.  For example, the quote above is often incorrectly originated to the famous Oscar Wilde (circa 1880) and his ironic sentence was that “imitation is the sincerest form of flattery that mediocrity can pay to greatness.”  However, earlier forms of this wording can be attributed to Charles Caleb Colton in his 1824 book (“imitation is the sincerest of flattery” [missing “form”]) and Eustace Budgell in a 1714 newspaper (“imitation is a kind of artless flattery”).  So it seems that Wilde may have used the first part of this quote from other sources and not given them credit, but clearly benefitted from this idiom.  He added the second part about “that mediocrity can pay to greatness” and thereby changed the intent of the phrase, but is this plagiarism?

Plagiarism is a serious offence in academe.  As an example of possible ramifications to plagiarism, consider my friend Karl Rohnke.  Karl was a pioneer in experiential education.  He developed thousands of activities, games, and exercises to enable learning.  Most of these ended up in his various books, where he often mentioned the names of those who had shaped his thinking or improved his methods.  However, others copied the content of his books without asking.  Many authors were from a foreign land and were genuinely offering him a tribute from their cultures by translating his work, even though they received the royalties and the author credit.  Others were from the same country as him (USA) and should have known better.  One American author erroneously received the Karl Rohnke Creativity Award from AEE (the Association for Experiential Education) for a book full of Karl’s past creations and AEE even asked Karl to present the award!  I talked with him about this and realized that he had simply stopped creating and sharing his ideas.  He had given up as a result of this plagiarism and the experiential profession suffered for his lack of contribution. I worry that the same can and will happen in universities and this will lead to a drop in creativity and innovation. 

Plagiarism must have consequences.  As an administrator, I was the final word on consequences for students who plagiarized.  For those new to academic life, plagiarism can be intentional or accidental.  Intentional plagiarism was deceptive and I saw it as grounds for expulsion from the course or dismissal from the institution depending on the severity of the deception.  Accidental plagiarism was usually due to careless writing or lack of attention to detail and I saw it as grounds for receiving a grading reduction or zero on the assignment, but with the chance to stay in class and improve on the next assignment.  In an effort to help students understand plagiarism and learn to avoid it, I also developed this flowchart and encouraged them to use content checkers like TurnItIn before submitting their assignments for grading.



CRITICAL THINKING AND SKEPTICAL SCRUTINY: I’ve noticed a number of cultural differences between Canada and the United States.  I base my observations on having spent about a third of my life in each country and a third in other nations.  While living, working, teaching, and travelling throughout Canada and the US, I have detected a pronounced difference in the critical thinking of their two peoples.  In summary, Americans appear to think less critically than their international counterparts.  I believe the reasons for this stem from not teaching critical thinking in the schools and universities to the extent that it is taught in developed nations outside the USA.  I wonder whether critical thinking is valued or desired in American culture.  I understand that an American population not thinking critically is more likely to believe its government, fall for false advertising, deny science, and fail to question religious dogma than those who embrace rational skepticism elsewhere in the world.  I know with reasonable certainty that this considerable difference in critical thinking partially explains recent American election outcomes.

Critical thinking derives more from critiquing and less from criticism.  It critiques your own thoughts and the ideas of others.  Critical thinking involves the seamless combination of problem solving, decision making, and sound judgment with objectivism, rationalism, logic, and skepticism.  It requires that the thinker question: purpose, assumptions, perspective, evidence, inference, implication, cognitive processes, background concepts, breakdowns in problem solving, biases to decision making, and heuristics for sound judgment.  While critical thinking uses higher order learning objectives from the Taxonomy, these are not the sole recipe as often presented by many educators.  In other words, higher order thinking is not enough to by itself to be critical thinking.  Not only does it evaluate evidence or arguments and identify errors and shortcomings, but critical thinking also evaluates and improves one’s own cognitive processes as a consequence of its practice.  It clarifies beliefs and enhances actions. It seeks to identify biases in research studies and social media, especially those that go unacknowledged by the authors.

In my experience, working in higher education as an administrator and teaching undergraduates as a professor in a dozen countries, I can safely say that American students think much less critically than those in other leading nations.  One explanation for this lies in the American secondary school system that fails to teach critical thinking as a mainstay (as it is commonly taught in Canada).  Most university courses and degree programs avoid critical thinking in the US, unless the professional discipline is one that requires it such as medicine, science, or engineering.  The outcome is an undereducated populace (even those holding college degrees) who are unable to think critically.  They buy whatever celebrities pitch, embrace conspiracy theories, and agree with their government or religious leaders.  The masses are easily polarized by consuming social media posts without question and passing these along as truths to others who think similarly in support of their fixed opinions.  They don’t stop to consider dissenting opinions and do not apply skeptical scrutiny to the information presented for their consumption.

Skeptical scrutiny means not blindly accepting information as true, but subjecting it to a series of several dozen questions designed to critique the information and determine its accuracy.   Those questions can be categorized under the typical titles listed: who, what, where, when, why, and how.

This is how I have been teaching critical thinking since 1985 when I was the teaching assistant for a research professor who asked me to give a lecture on critical thinking and the critique of research.

  1. The critical thinker considers the eight evaluation criteria for the idea being presented and repeats some criteria that apply to the individual components of the idea or its thinking.
  2. The eight thinking components, which usually combine to make the idea presented, are examined in detail for their contributions, strengths or weaknesses with respect to the idea.
  3. Finally, the idea and its components (or any other related aspects) are subjected to a series of skeptical scrutiny questions to determine the ultimate truth, validity, and reliability of the idea.

This trifecta of skepticism has to be balanced by continual openness toward novel ideas and the humility to accept that your thinking might be erroneous and in need of a change.  Without either, personal growth will be next to impossible.



TEACHING AND LEARNING HOW TO DO RESEARCH: I've taught a lot or research courses in various social science disciplines at many institutions. Based on those, I have evolved an instructional sequence that I liken to a learner's ascent of an escalator and I am aware of the irony that research students often feel like the escalator has stalled or is even going backwards for them! Nevertheless, I find this experiential sequence of four courses has proven to be useful and effective for individual graduate students and undergraduate teams of learners alike.

  1. CONSUMPTION of research requires critical thinking. In this first course, I explain plagiarism and frequently assign lots of reading and writing in the spirit of the Liberal Arts. I enable students to think critically with a healthy dose of skepticism and provide them with increasingly flawed, obscure, and complex studies to review. Once they know how to critique, I get them writing essays that synthesize (after critiquing) several studies into a singular and well-stated point. Finally, I have them complete a literature review on their choice of the perfect trifecta of subjects, constructs, and treatments.
  2. PRODUCTION of research starts with program evaluation. In this second course, I like to have the class conduct a variety of program evaluations for non-profit groups on campus and in the community.  This wide range of studies provides a robust rainbow for others to learn from as they conduct their own study. This is the time to introduce ethical considerations, methods of data collection, and descriptive statistics (with maybe a few inferential ones, if the learners are ready). I make sure that students collect reasonable amounts of qualitative and quantitative data so we can focus on the analyses of both. Finally, we practice writing reports that the customer will comprehend and appreciate.
  3. EXECUTION of research examines the depth and breadth of inquiry. In this third course, once students are well grounded in the basics from their course in program evaluation, I have them conduct a campus or community-based research study. After explaining the depths of designs, hypothesis testing, sampling of populations, and other key research factors, I ask them to repeat the trifecta exercise, where they select their subjects, constructs, and treatments for their study. With as little help from me as possible, they find their own sponsors (some small amount of funding will be needed to cover their expenses and I tend to know where to access these sources). Again this presents a diversity of studies, where students can learn from one another as they conduct their own study. Next, they learn about instrumentation and other means of gathering and measuring data. During their break, between this penultimate course and the ultimate one in the sequence, they collect their data.
  4. ANALYSIS should include a balance of qualitative and quantitative data.In this fourth course, students learn to code and analyze their qualitative and quantitative data. Once again, everyone is sharing their progress, difficulties, surprises, and data experiences within the class.  I concentrate on what they should and should not write in their discussion of their analyzed findings. After learning parametric and non-parametric inferential procedures, and the analysis of common content in data from comments, interviews, and observations, they repeatedly write drafts of their final report, but with an eye and ear toward publishing and/or presenting their results.  Students practice their presentations and share their publications with plenty of peer feedback in both instances and submit or present their final products.  In the end, they are well prepared to learn how to write a formal proposal for their next project, study, thesis or dissertation.



CYCLE OF SCIENTIFIC INQUIRY: Scientific inquiry is a repeating cycle of procedural techniques used to investigate noticed incidents or perceived occurrences (called phenomena).  This initial search creates new knowledge (replacing or modifying existing knowledge) and leads to new investigations with cycle repetition (called RE-search).  The exploratory or confirmatory cycle is conducted with rigor, employs logical reasoning, may control for influencing factors, and is based in empirical evidence, hence making the inquiry scientific in nature.  Fixed in the scientific method and a process for educational research with humans as the subject of study, the inquiry sequence is as follows.

  1. CURIOSITY generates a question, but only after becoming aware of a certain phenomenon
  2. A PRIORI  synthesizes what is known by reviewing literature and other existing information
  3. THEORY formulates a hypothesis that is preferably expressed in the “null” or testable form
  4. PREDICT outcomes by explaining expectations and considering all personal influences (biases)
  5. PREPARE the study by precisely and meticulously designing the study plan of six elements:
    • DEFINITIONS – delineate key terms and scope (what is included/excluded from study)
    • METHODS –type of study (experimental, historical, observation, interview, action, etc.)
    • SUBJECTS – sample the population (randomly or purposefully represent the whole)
    • CONSTRUCTS – identify the variables (decide how to measure the ideal experience)
    • OBSERVATIONS – choose instruments (the tools or ways to measure and collect data)
    • MEASUREMENTS – when to administer (once, before and after, over many times, etc.)
  6. PERFORM or conduct the study by accurately carrying out and verifying these four segments:
    • COLLECT – gather the data (these data may be quantitative, qualitative, mixed, etc.)
    • PREPARE – code and enter data (address errors and missing or duplicated values)
    • ANALYZE – examine database to get findings (using statistics, common content, etc.)
    • INTERPRET – explain meaning of findings (clarify numerical outcomes or descriptions)
  7. DISCUSS or suggest relevance of findings in relation to prior background, theory, and prediction
  8. CONCLUDE or summarize results (findings + discussion) so as to answer the original question
  9. RECOMMEND or propose next study cycle and changes to human behavior or knowledge base
  10. CRITIQUE by a peer review that identifies study flaws, confirms results, and improves next study

These 10 steps, divided into equal halves (proposing and completing the study), fit well with teaching graduate students and novice faculty members how to conduct research.  While not a perfect sequence, since some steps will need adapting and modifying to suit unique types of research, this serves as a reasonable guide to scientific inquiry.



THE FAMILY TREE OF STATISTICS: When I was working on my doctorate, I had the great pleasure to support Lorraine Davis, teacher extraordinaire of statistics. I learned a great deal from her, but the greatest gift she gave me was the opportunity to be her teaching assistant and a lead instructor of the lab sessions that complemented her lectures. This forced me to know my statistics really well and struggle to be as good as her at teaching them.

While working together on a small handbook of statistics in 1985, we evolved this family tree to show how some of the more basic tests related to one another. The arrangements below are rather self-explanatory.



DATA FORMAT AND DESCRIPTIVES: Once learners understand the four basic data formats, they will know how to statistically describe these (using central tendencies and variabilities), as well as the correct inferential procedures to apply (parametric or non-parametric). A summary of these possibilities is shown below. The four data types are:

  1. NOMINAL: qualities expressed in categories and named or labeled according to distinct attributes. Examples include: gender (male, female, other, etc.), hair colour (grey, blond, brunette, redhead, bald, etc.), religion (Islam, Christianity, Judaism, etc.), and nationality (Canadian, British, Australian, American, etc.). This categorical format cannot be meaningfully sequenced and the distance between values will not make reasonable sense.
  2. ORDINAL: qualities arranged in sequence, but where the distance between values cannot be determined. Examples include: finish positions in a race (first, second, third, disqualified, etc.), spicy foods (mild, hot, hotter, hottest, deadly, etc.), and letter grades (A, B-, C+, D, F, etc.).  Ordinal data cannot be converted to interval data, because the distances between values are unknown.
  3. INTERVAL: quantities arranged as continuous numbers that have equivalent differences among values. Examples include: age (-3 months, 27 years, etc.), distance (10.4 km, 56.89 mi, etc.), BC/AD dates (2637 BC, 1958 AD), and Celsius/Fahrenheit temperature (-40 degrees is the same point for both scales), where these can have negative values below arbitrary zero points. This continuous format could be sequenced (data would become ordinal), but is rarely required.
  4. RATIO: quantities arranged as continuous numbers that have an absolute zero and equivalent distances between values. Examples include: height, weight, years of education, and Kelvin temperature, where these variables cannot have negative values below zero. To simplify matters, treat ratio data as interval for all statistical procedures.



DECISION SEQUENCING FOR INFERENTIALS: Selecting the correct inferential procedure from a dozen choices of specialized statistics can be extremely daunting.  I prepared this decision sequence to help undergraduates navigate the myriad of options available to them.  In order to use it effectively, one must know:

  1. Whether the variable data are discrete or continuous
  2. What the data format is for each variable
  3. Whether variables are being correlated or predicted
  4. How many condition, groups, or samples (CGS) are being analyzed
  5. If a sample is being compared to a population
  6. Whether the population standard deviation is known or unknown
  7. If two samples are being separated/differentiated
  8. Whether the CGS are independent or connected on a commonality
  9. How many independent variables (IV) are being analyzed
  10. How many dependent variables (DV) are being analyzed



A CONFIRMATION MATRIX FOR STATISTICS: Something Lorraine shared with me, early on in our conductor passenger relationship, was a basic confirmation matrix as a double check to make sure one is using the correct inferential statistical test. I expanded it to its present content in 1986 and have applied it over the years since then. The key to using this matrix is knowing your data type (as above). The format for both independent and dependent variables can be: interval, nominal, and/or ordinal. When necessary, interval data can be converted to ordinal data, but not the reverse. Nominal data cannot be converted.Simply find the corresponding list of tests from both data formats and then triple check against the family tree earlier.



THE INSIGNIFICANCE OF STATISTICAL SIGNIFICANCE: If you could accurately measure the height of every adult on the planet and compare those averages for males to females (assuming these were the only two genders), then you would not require statistics to determine which gender was tallest and by how much on average, because you would have measured the entire population and determined the exact difference.  No need to conduct a t-test; no concern with probabilities.  You would have 100% certainty.

However, measuring everyone is impractical and expensive, so we must take a sample instead.  We do what we can to ensure that the smaller sample is representative of the larger population by applying various methods, such as random selection or proportionate characterization.  For example, when measuring the height of adult Canadians, we might get an alphabetical citizenry list (22 million in 1971), and select every ten thousandth citizen starting with a random number between one and ten thousand.  Then, we might check that this collection of (n=2,200) subjects was equally represented across race or age groups with the same percentages of races or age groups in both sample and population.  We do this to strengthen the representation of the population by the sample we accept as accurate.  With strong representation, we can be more confident in generalizing our sample findings to that population.

We also want each variable (the way we measure something) to be representative of a construct (the thing we want to measure).  For example, if we want to measure heights and weights (to see if an association or correlation exists between these two constructs), then we must be certain that the ruler and scale (used to obtain two variables in centimeters and kilograms) are both valid (measure what they claim) and reliable (measure consistently and without error).  If we use a more than one of each of these devices in various locations, they must all be calibrated against one another or a third party standard.

Despite our best efforts, any design will be flawed and so we need statistics to not only tell us whether things are different or related, but also how sure we are compared to 100% perfection.  So we test null hypotheses: no differences and/or no relationships.  We get a t-value or r-coefficient of correlation and compare that to a standardized table of values to determine whether or not the number we obtained exceeds the threshold we set as alpha=0.05 (or sometimes 0.01), based on 95% (or 99%) confidence levels. 

The resulting p-value is our statistical significance.  It is a statement of certainty about whether the difference or relationship found was due to chance, study design flaws, and/or pre-existing conditions of the groups studied.  It tells us how much we can be confident with and rely on the outcome as true.  It does not tell us whether that difference or relationship is meaningful or has a magnitude of importance.

If we found significance, then we report a difference/relationship existed, and discuss the two groups separately (difference) or two variables as associated (relationship.  If we didn’t find significance, then we report no difference/relationship was found (but NOT “no difference/relationship exists” – this is an important distinction) and discuss the two groups together as one entity (no difference) or unconnected with each other (no relationship).  To claim no difference or relationship exists, we would need to do further statistical calculations.  Unfortunately, consumers of research findings, most often the non-scientific media, erroneously report none exist, when the researchers simply said they failed to find any.

Also, the outcome can simply be wrongly determined as a false positive or a false negative.  In a false positive (type I error), the null hypothesis is incorrectly rejected.  In a false negative (type II error), the null hypothesis is incorrectly not rejected (term “accepted” is never used).  Reducing the chances of making one error increases the chances of making the other.  For example, the North American legal system is designed to avoid type I errors (false positives), such as convicting an innocent person, at the cost of making plenty of type II errors (false negatives), such as acquitting guilty people.  The same can be said for social science: we would prefer not to find something that exists, and hopefully find it later on in future research, rather than find something that isn’t there, and mislead current practice.  This conservative approach, coupled with the drive to publish significant (p< α) results, has led to a deluge of publications that failed to emphasize practicality and overlooked important, but insignificant research.



THE IMPORTANCE OF PRACTICAL MEANING: In my early days of teaching advanced statistics, students had previously been taught to obsess about statistical significance (above).  They had been told to seek p<0.05 (or in some cases p<0.01).  Their previous instructional faculty were only getting published by finding such statistically significant outcomes.  Very few journals would publish non-significant results.

As noted, the p-value is only a statement of certainty that the outcome of a statistical test (t, r, F, etc.) has only a slight percentage chance of being incorrect.  It doesn’t mean the differences or relationships are quantified, only that they exist in that instance.  Obtaining a significant finding is as easy as simply getting more subjects, since p-value computation and verdict are based on the sample size studied.

Practically meaningful results should be more important than statistically significant findings.  For example, take the typical well-controlled experimental research study design, where you have two groups.  Group C is the control group: they receive no special treatment during the study.  Group E is the experimental group: they receive the special treatment.  Both are tested before and after the treatment period and live their lives in parallel during the study.  This allows for other factors that may influence the outcome to play upon both groups equally.  We can tighten the control of this study, by ensuring that the members of each group are matched on a number of characteristics.  For each subject of a certain gender, age, race, etc. in group E we have a subject in group C with identical demographics.

When the study is complete, we would typically conduct an ANOVA to compare the results of the four measurements (C-before, C-after, E-before, and E-after).  In a hopeful study, groups C and E would not be found to differ on their pre-tests, but E would be statistically different than C on post tests, thus indicating that the treatment worked and we would have a nice p-value to go along with the F-value.  In fact, we used ANOVA (comparing everything together), instead of several matched t-tests (comparing the four measurements separately), because ANOVA is much more likely to find a significant outcome.

The resulting F-value tells us that at least one of the four measurements differs from the others and we can use “post-hoc” analysis to determine which.  These tests also bring significant p-values.  However, all this battery of tests will tell us is that a difference was found and nothing more.  They will not tell us the amount of the difference (we need to consider the descriptive means for this and the use of the effect sizes described below).  Nor will they tell us if the outcomes are worthwhile.  For that, we must examine the practical value of what we find and this requires us to consider several aspects: improvement, cost, context, and confidence interval.  Failure to do so makes our research meaningless.

The first, improvement, refers to whether or not the finding is a noticeable enhancement over the status quo.  For example, if we take ferries to and from an island and have two different routes to choose from, we can measure the average crossing time, including the commute to and from the terminals at both ends, and determine which is the fastest.  If we find a statistically significant five minute difference, we can rejoice in the p-value.  However, once we consider practicality, the celebrations may cease.  If the overall times are ranging from 25 to 35 minutes, then 5 minutes can be a great savings, but if the times are measured in several hours, does 5 minutes really make that much of a noticeable difference?

The second, cost, considers whether the improvement is worth it or not.  In the ferry example, 5 minutes faster is a definite step up for a half hour trip, but is it worth the cost, if the faster ferry has a $100 higher ticket?  How about $10 more?  What about $1?  Sometimes, it’s a personal choice.  What happens if an amazing new drug lowers systolic blood pressure, by 30 points on average, but each pill costs a million dollars?  How does that compare with other anti-hypertensive drugs already on the market?

The third, context, judges the outcomes and relevant within the circumstances they were studied and not only includes generalizability, but also, and more importantly, excluded situations.  How does weather impact the ferry times?  Does the 5 minute difference hold up in all seasons and times of day?  Did the hypertension study only consider patients with extremely high blood pressure to begin with?  Would the price of the drug be expected to come down over time as more and more pills were manufactured?

To complicate matters, the 5 minute improvement in the ferry example is not an accurate number.  It was the estimated difference between two means (or estimated mean of many differences with the blood pressure drug) and as such, we need to calculate confidence intervals to insure that the mean we came up with is a good estimate of the population average.  So the fourth, confidence, is the measure of precision that a population average falls close to the sample mean (the unknown amount is within our estimated range for it).  In short, this accounts for a margin of error found in all statistical procedures.

To calculate a confidence interval, we use the standard error (not standard deviation) for a distribution around the mean we have estimated.  This value is called 1 Standard Error Unit (SEU) and we base our calculations on the “normal” curve.  Two SEUs are typically applied in both directions above and below the mean for a 95% confidence interval.  We use 3 SEUs above and below for 99%.  These most common intervals are based on our original probability settings of alpha<0.05 or <0.01 respectively.

The confidence interval is expressed as a range with a percentage.  The difference between both mean ferry times is 5 minutes ±2 minutes at 95% confidence.  This means the ferries might actually travel in a couple of minutes more or less than the estimated gain.  The new anti-hypertensive drug lowers systolic blood pressure ranging from -10 to +70 points on average with 99% confidence.  Sometimes, these new numbers can place the possibility of no difference inside the calculated confidence range.  In other words, the drug might really raise blood pressure in a few cases and make no difference in others.   Now, has your thinking changed on how practically meaningful the significant findings are for this drug?



THE OVERLOOKED VALUE OF EFFECT SIZES: Effect size is one way to represent practical meaning.  It computes the strength or magnitude of the difference or association that was discovered earlier using inferential statistics like a t-test or correlation.  To me, effect sizes are more valuable than probabilities.

Initially, effect size can seem unfamiliar and therefore scary because you probably didn’t learn about it in your statistics education.  However, it is a fairly simple concept to understand and easy to calculate.

First, I made an earlier point about claiming differences were “not found” as opposed to “didn’t exist” because that is what that statistic determines.  In order to claim the latter, you need to understand whether the circumstances behind the test (sample size, alpha level, etc.) were powerful enough to find that difference if it did indeed exist and perform a power analysis to calculate a β-value.  A β-value is the probability of a type II error or false negative (remember that α-values were the same for type I errors or false positives).  Test power is expressed as 1-β, where β is calculated by several complex formulas. Low β-values suggest powerful (1-β) tests and we can finally conclude with "it/these didn't exist."

Calculating effect size is a simpler approach and, although almost 100 diverse formulas exist, several of the computations are quite easy and relate to the overlap of distributions (as in t-tests or correlations).  Like most statistical procedures, these effect size formulas are organized by differences and association as in the diagram below.

For differences, “Cohen’s d” is the standard.  By taking the difference between two means (or the means of differences), and dividing either difference by the pooled standard deviation for both distributions, a precise effect size can be obtained.  Different statisticians have suggested a plethora of ways to calculate pooled standard deviation.  The simplest involves adding two calculations (squared standard deviation or variance times degrees of freedom or sample size minus 1) one for each distribution.  The sum of these two calculations is then divided by the sum of both degrees of freedom (n1+n2 -2) and then the square root is taken of the resulting division.  However, for ANOVAs (measuring differences across multiple groups and/or variables), “eta-squared” (sum of squares for effect divided by sum of squares total) and “omega squared” (sum of squares for effect minus degrees of freedom times mean squared for error and then divided by sum of squares total added to mean squares error) are better formulas.

For associations, “Pearson’s r” is the standard.  By squaring the r-coefficient, a measure of explained or shared variance is obtained and depending on the size of this number (expressed as 0-100%) a claim can be made about the strength of the relationship.  However, for multiple regressions, “Cohen’s f2” is the better formula (ratio of explained variance = R2 to unexplained variance = 1-R2).  When data are nominal in format (as in a Chi-square), an “odds ratio” (observed odds divided by expected odds) is the standard.

The following diagram organizes a few of the more commonly used formulas for estimating effect size by simple or complex and by difference or relationship divisions.  Many more formulas exist than these few.  When reporting estimations of effect sizes, please be certain to describe the formulas that you used.

I can remember my first few p<α publications, where well-meaning, but uneducated editors changed confidence intervals to significance levels, dropped standard deviations from tables (because “they take too much room”), and refused to publish effect sizes (because “no one really cares about those”).  The absence of these values prevents other researchers from performing very important comparisons.  I quickly learned to seek statistical significance above all other important considerations.  These days, I no longer publish much in the way of peer-reviewed journal articles, but when I do, I maintain a passion for including a minimum of the effect sizes.  Luckily, journals are slowly changing to accept these statistics, because these numbers perpetuate further research by others and the bbody of knowledge can become cumulative.



CONDUCTING META-ANALYSES: Meta-analysis is a statistical procedure that produces a singular synthesis of total evidence, where each subject being studied is not an individual human being, but the entire research study itself.  Ideally, it integrates multiple and related effect sizes (explained above) from similar past studies into a collection of evidentiary statements that inform future practice.

Meta-analyses typically consider studies that are similar on a minimum of three out of the four usual PICO factors: Participants (subjects’ characteristics), Intervention (treatment applied or action taken), Comparisons (before and after or control and experimental), and Outcomes (changes in variables or constructs measured).  Some researchers refined their pool of studies by considering the dates of publication (within a range of years) and comparable study designs (randomized trials, cohorts, etc.). 

One big strength for meta-analysis is that it provides a considerable gain in statistical power (the ability to quantify impact), because combining studies increases the overall sample size.  This is especially useful when the past studies being analyzed have small sample sizes or weak findings.  However, it has two weaknesses.  It is criticized for comparing apples and oranges, especially when less than three PICO factors are similar.  It contains a publication bias, because studies with statistically insignificant findings were unpublished and this over-estimates a final effect size by omitting smaller contributory effect sizes.

The meta-analysis builds on top of a systemic review.  A systemic review differs from a typical review of the literature in terms of linkage.  A typical review of the literature summarizes all the previous studies that relate to the new research being conducted by considering similar methods, designs, questions, purposes, subjects, samples, populations, variables, constructs, and anything else that seems relevant.  However, these previous studies don’t have to be connected in anyway.  For example, if I want to use a unique procedure in my new research, I only have to review studies that used that same procedure, even if these studies were from entirely different disciplines from mine.  On the other hand, a systemic review connects the past studies under review by common PICO factors and so these must be examined for quality and linked to one another.  The systemic review forms the initial 8 steps of a meta-analysis.

  1. Identify PICO factors: What participants, interventions, comparisons or outcomes are you most interested in?  Is there a particular design you seek or wish to avoid?  Do you have a publication time period in mind?
  2. Frame a research question: Begin with a loose free-form query created around your PICO interests; explicitly tighten and structure this question as more information becomes available.
  3. Set a criterion number of PICO factors: Usually 3 out of 4 is a reasonable minimum, but this can be deliberately decreased or increased, if and when an abundance or scarcity of studies is discovered during the next search step.
  4. Search for prospective studies: With the PICO factor criterion in mind, find every study that fits the eligibility by searching with Boolean logic (and, or, not), fuzzy strings (near, within, partial wording, letter transpositions) and wildcard (*) or truncation ($) terms where useful.
  5. Create a rejection scheme: Make a list of reasons to reject a study and then build a checklist for the evaluation of each study from those reasons (answers other research question, doesn’t fit PICO, published outside time period, inappropriate design, can’t translate, not peer-reviewed, poorly explained, contains obvious flaws, etc.)
  6. Examine the quality and linkages of the study pool: Read titles and abstracts first in order to eliminate those studies that don’t meet the reasons above and only read more of the study (conclusions, results, methods) if inclusion or exclusion is not obvious from the abstract.
  7. Retain the qualifying studies: Keep those studies that meet your criterion for PICO factors and exhibit high quality regarding the reasons, then toss out the rest (keep a record of these eliminations with accompanying explanations for later use in writing up the meta-analysis). 
  8. Fully read the retained studies: The above couple of steps are repeated with each entire study, not just its title and abstract to ensure that it is correctly included for meta-analysis (extra work investigating citations or referrals may be necessary to enable a complete understanding).
  9. Note statistical content: track the presence of statistically generated numbers within the study (number of subjects, means, standard deviations, t-values, R-coefficients, F-values, probabilities, confidence intervals/standard errors, etc.) and data formats (interval, nominal, ordinal, etc.).
  10. Select type of effects: Choose between fixed or random effect meta-analysis models based on information obtained in the prior step above.  Fixed effect assumes one true effect size for all the studies and the same singular population, where different effect estimates are due to sampling errors.  Random accepts that effect size may differ across studies that sample a wide variety of subject characteristics and so different effect estimates are due to that variability.
  11. Estimate effect sizes: Use the appropriate formula for the studies’ type of data, statistical tests, and effects models suggested so far.  For example, employ “Cohen’s d” for differences, “determination coefficient” for relationships, and “odds ratio” for nominal (see diagram above).  Many other tests exist to choose from, but be certain your decision is based in the study content and context.
  12. Prepare a forest plot: Chart effect size as a box (where the box area is inversely proportional to the effect size variance) with horizontal confidence intervals parallel to the x-axis. Order these along the y-axis according to their publication date.  In this blobbogram, compare the boxes and intervals for overlap across a line representing “no effect” fixed on the x-axis (set at 1 for ordinal odds ratios and 0 for interval differences or relationships).  If this number falls within the confidence intervals, we can claim no effect found, but if the confidence intervals are outside of this line, then we can claim a definite impact.
  13. Check for publication bias with a funnel plot: Graph effect size (x-axis) against sample size (y-axis) in a scattergram and look for the shape of a funnel in the data points (one per study) for separate variables.  An inverted funnel-shape indicates homogeneity of studies and a lack of bias, while an empty lower quadrant suggests heterogeneity and the presence of possible bias. due to the lack of unpublished studies (perhaps without significant findings) that could make a valuable contribution to effect size.
  14. Conduct secondary analyses: Examining subgroups by meta-regression is warranted in cases where an obvious progression is present (studies on subjects of varying ages or illness levels).
  15. Present and interpret the findings: Finally, write up the research in accordance with normal publication procedures, but be prepared to make more definitive statements than usual, because the meta-analaysis permits and potentiates these.



QUALITATIVE EQUIVALENCIES OF META-ANALYSIS: The Bible, Qur’an, Sutras, Vedas, Talmud, I Ching, Kojiki, and Granths are examples of the many religious chronicles that describe a sequence of legendary events and the widely applied metaphoric messages and life lessons within them.  These are collectively co-created and widely well-read illustrations of a qualitative equivalent of the statistical meta-analysis.

Just as meta-analyses combine and integrate the quantitative data of many closely-related and similar past studies, nine methods are available to synthesize and synergize the qualitative counterpart to this.  These include: meta-ethnography, grounded theory, thematic synthesis, narrative synthesis, meta-study, meta-narrative, critical interpretation, ecological triangulation, and framework synthesis (Barnett-Page & Thomas, 2009).  For the most part, these approaches follow the steps of a systemic review (above) and then add unique methods of qualitatively synthesizing the findings of the reviewed studies.

Just as a quantitative researcher needs to know the correct formulas to apply to numbers in a meta-analysis, the qualitative researcher must be able to choose the appropriate method for the synergy of characteristics, context, and content, while understanding the syntheses produced by all nine methods. Since no censensus exists as to the best method, I offer this summary and personalized selection process.

  1. Meta-ethnography emphasizes the reciprocal translation of concepts and metaphors across homogeneous studies.  The approach merges these concepts and metaphors into a single interpretation that preserve the cultural context by multiple iterations during synthesis.
  2. Grounded theory is interested in explaining social phenomena through the creation of a new hypothesis anchored in the data of homogeneous studies.  The approach codes content into categories and tracks the evolution of theory through constant comparison during synthesis.
  3. Thematic synthesis aims to generate higher-order patterns of meaning from heterogeneous studies that have been stringently chosen.  The approach codes important or recurrent themes and connects these on commonalities with some iteration during coding and synthesis.
  4. Narrative synthesis focuses on the words and textual content from heterogeneous stories that have been rigorously selected.  This highly subjective approach to conceptually map a great story may potentiate bias through analysis procedures that can appear loose or hidden.
  5.  Meta-study is sometimes used as a catch all phrase.  However, its purpose is to examine heterogeneous study elements (theory, methods, analysis, and synthesis) independently from one another.  The approach collects these four elements into a new combined interpretation.
  6. Meta-narrative intends to uncover the unfolding dimensions among heterogeneous studies by continually iterating at all stages.  The approach focuses on the sequence of events that form the storyline and distills these along with anecdotal tales into a single mapped grand narrative.
  7. Critical interpretation is an off shoot from meta-ethnography.  The approach uses iteration in all stages of the systemic review of heterogeneous studies, including refining research questions, searching for literature, selecting studies, applying codes, and defining categories.
  8. Ecological triangulation has the strictest iteration and criteria-based quality control for study inclusion across these nine methods.  The approach examines the studies from a variety of viewpoints and pulls apart interdependent relationships, reapplying these in specific situations.
  9. Framework synthesis builds structure from heterogeneous studies that have been meticulously included by iteration only during the selection process.  The deductive approach creates a conceptual model, but can be improved with iterative analysis to modify the framework.

These nine can be ordered by epistemology (how one chooses to understand the nature of knowledge) and ontology (how one chooses to comprehend the nature of reality).  Imagine a continuum ranging from naturalistic through pragmatistic to positivistic as shown by both red arrows on the triangle below.  The nine methods are distributed with the meta-methods in close proximity to naturalistic beliefs, while the synthesis methods are near the pragmatistic philosophy.  Only triangulation is near positivistic ideas.

From here, the nine methods are further teased apart by three factors: use of quality criteria to exclude studies, use of multiple iterations throughout the synthesis of qualitative data, and a focus on accepting heterogenous rather than homogenous studies.  The resulting dispersal is diagrammed below in a sort of qualitative Discriminant Function Analysis.  This dispersal allows qualitative researchers, with a clear understanding of their own philosophies of inquiry, to select the best method for their synthesis needs.

Barnett-Page, E. & Thomas, J. (2009). Methods for the synthesis of qualitative research: a critical review.  BMC Medical Research Methodology, 9, 59.  Retrieved Dec 9, 2009.



USE SLIDERS FOR TRULY INTERVAL DATA: Note the importance of interval data in the statistics matrix and family tree above.  The most potent statistics require interval data (continuous numbers) and this is the highest quality of data format.  Unfortunately, when we use Likert-like scales (subjects rate something on a scale from 1=low to 5=high) we restrict responses into categories (in this simple example, subjects only have 5 choices). These data, arranged categorically, are nominal in format (subjects can’t respond between the numbers) and suggest less potent statistics must be used.  However, since the categories are ordered in a line, we inappropriately "fudge a bit" and say the data are interval enough, when they really are only nominal, and incorrectly use the most potent statistical procedure.

Concerned with the purity of data in 1990, I began using scales that allowed subjects to place a mark on a continuous line ranging from 0 to 100.  In this way, the data were much likely to be interval than nominal.  At virtualteamworks.com, we started collecting data automatically by 2000 by using sliders like those shown below.  This example (demonstrated here for a tourism topic that the average person can relate to) is called a semantic differential, because it asks the subject to distinguish between two words that are opposite adjectives. TRY IT!

  • You will be presented with ten pairs of opposite words to complete the same sentence about your travel choices.
  • Move each slider square to the position that best represents your preference between the two opposite words.
  • If you absolutely prefer one word over another, then put the slider square all the way (100%) over to that side.
  • If your preference is balanced between the two words equally, then leave the slider where it is (50%-50%).
  • When you have positioned all 10 slider squares where you want them, click on the SUBMIT button. Thank you.
Statement: I like my tourism experiences to be...


% Relaxing % Exciting


% Passive % Active


% Entertaining % Educating


% Consuming % Conserving


% Familiar % Novel


% Safe % Risky


% Group % Solo


% Easy % Difficult


% Escaping % Searching


% Crowded % Deserted


  • When finished, please click on the submit button below. Thank you for taking the time to complete this suvey
  • Your responses will be anonymous and confidential. Any data reporting will be aggregated over the sample.


RESEARCH INSTRUMENTS: Here is a collection of instruments that I authored or co-authored in the 1990's.

General Surveys


Interpersonal Trust

TEAM EFFECTIVENESS ASSESSMENT MEASURE: LONG (100 items, 30 minutes, single use) & SHORT (50 items, 15 minutes, repeated use)
VIRTUAL TEAMWORK (50 items, 10 minutes): SELF (use for self) & OTHERS (use for team members)
measures INTERPERSONAL TRUST on 6 dimensions:
D= Dependability,
E= Encouragement, and
O= Overall Trustworthiness
LEADERSHIP EFFICACY ASSESSMENT DIAGNOSTIC: LONG (100 items, 30 minutes, single use) & SHORT (50 items, 15 minutes, repeated use)
ELECTRONIC FACILITATION (50 items, 10 minutes): SELF (use for self) & OTHERS (use for team members)
PRIEST ATTARIAN RISK TAKING INVENTORY: BUSINESS (10 scenarios, 30 minutes, use pre- and post) & OUTDOOR (10 scenarios, 30 minutes, use during)
GLOBAL LEADERSHIP (50 items, 10 minutes): SELF (use for self) & OTHERS (use for team members)
DIMENSIONS of the ADVENTURE EXPERIENCE: LONG (26 items, 15 minutes, once daily) & SHORT (2 items, 1 minute, multiple times)
ONLINE TRUST (50 items, 10 minutes): SELF (use for self) & OTHERS (use for team members)
s i m o n _ p r i e s t @ y a h o o . c o m
CONTENTS: Copyright © 1975 | All Rights Reserved
Unauthorized use and/or duplication of this content without express and written permission from simonpriest.com is strictly prohibited.
Links to this content may be used, provided that full and clear credit is given to simonpriest.com with appropriate and specific direction back to this original content and page.