How UnPeeragogy Collects and Examines Evidence

A transparent, exploratory approach to bringing practitioner experience back into theory.

This approach shares lineage with realist evaluation's use of Critical Incident Technique to refine programme theory (Pawson & Tilley, 1997), applied here to an open, continuously revisable, publicly crowdsourced corpus.

Research question

What happens when Peeragogy patterns meet conditions they don't explicitly describe?

This question is deliberately open. It does not assume that patterns fail. It does not assume they succeed. It assumes that the conditions under which a pattern operates — group size, trust, commitment, time, shared history, material constraints — are not fully captured by the pattern description itself. When those conditions diverge from the unstated assumptions of the pattern, something happens. We want to know what.

Sub-questions include:

• Under what conditions does a pattern appear to work?
• Under what conditions does it become fragile?
• What unexpected outcomes emerge — positive, negative, or neutral?
• Which assumptions appear to be necessary but unstated?
• Which observations challenge the UnPeeragogy interpretation itself?

The last sub-question is structural: the framework we are building must remain open to its own revision.

What counts as evidence

We distinguish between different kinds of statements. This is the epistemic taxonomy underlying the project:

Source

What the Peeragogy Handbook says. This is fixed text — the original pattern description, principles, and claims.

Observation

What a practitioner reports from direct experience. Not yet interpreted — just what happened, as concretely as possible.

Incident

A specific episode described using the Critical Incident Technique (CIT). More structured than a general observation: it has context, intention, action, outcome, and reflection.

Interpretation

What the researcher derives from the incident. This is always provisional and linked to the specific incident that generated it.

Hypothesis

A tentative explanation of why a pattern behaved as it did. Not a conclusion — a candidate to be tested against further incidents.

Failure Mode

A describable way in which a pattern deteriorates or produces outcomes contrary to its intent. Failure modes are more stable than best practices because they have been observed under specific conditions.

Counter-Evidence

An observation or incident that puts tension on a previous interpretation. This is not necessarily a refutation — it may qualify, complicate, or narrow the scope of an earlier claim.

Revised Interpretation

An interpretation updated after new counter-evidence. The old interpretation is not erased — it remains visible as part of the audit trail.

These types are not decorative. They are reflected in how field reports are structured and how the knowledge graph connects entries. A reader should be able to see what kind of claim they are looking at.

Evidence status

Every observation and interpretation in the corpus carries an evidence status that indicates its current epistemic position:

We do not assign numeric confidence scores. These qualitative states are more honest — they describe what we know and how we know it, without pretending to measure validity on a scale we have not calibrated.

How incidents are collected

Field reports are collected through a public GitHub Discussions forum in the field-reports category. There is no formal sampling frame: contributions are voluntary, self-selected, and asynchronous. This means the corpus is not representative in a statistical sense — it captures what people choose to share, not a random sample of Peeragogy practice.

Two templates serve different entry points:

📖 Share Your Story

For anyone who tried something and wants to report what happened. No prior knowledge of the project's vocabulary is required. The form follows a natural sequence:

1. Context — What group? Size, purpose, duration?
2. Intention — What were you trying to do?
3. Action — What did you (or the group) actually do?
4. Outcome — What happened? Be specific.
5. Surprise — What was unexpected?
6. Reflection — Looking back, what do you think contributed to the outcome?

The form separates what happened from what the contributor thinks it meant. Both are valuable — but they are different kinds of evidence.

🔍 Structural Analysis

For contributors already familiar with the framework who want to propose a specific revision to an existing entry. This template requires position (Confirms / Extends / Contradicts / Question), a detailed analysis, and optional suggestions for impact. The final editorial decision rests with the project — this prevents the framework from outsourcing its judgment to whoever happens to submit a report first.

Critical Incident Technique (CIT)

Our primary collection method is adapted from the Critical Incident Technique, originally developed by John C. Flanagan (1954) for studying effective and ineffective behaviours in complex real-world settings. CIT asks respondents to describe a specific incident — not to offer general opinions, and not to confirm or refute a theory.

Why CIT, not AAR: After Action Reviews assume a shared team experience and a common goal. Most Peeragogy field reports come from independent observers in different contexts. CIT is designed for exactly this situation: asynchronous, context-rich, single-observer reports.

What CIT can do:

• Generate detailed, concrete descriptions of real events
• Identify conditions that affect outcomes
• Surface unexpected patterns and contradictions
• Provide the raw material for inductive analysis

What CIT cannot do:

• Produce statistically generalisable results
• Establish causal relationships
• Estimate prevalence of a failure mode in the population
• Eliminate recall bias or narrative self-justification

We do not treat CIT reports as objective accounts. They are retrospective narratives shaped by memory, perspective, and the contributor's relationship to the events. This is not a weakness to be eliminated — it is a characteristic of the data type, and we account for it in our interpretation.

Avoiding leading questions: The form does not ask "did this prove the pattern wrong?" because that question already contains an interpretation. It asks what happened, what was expected, what surprised — and lets the friction (or lack of it) emerge from the description, not from the framing.

From incident to interpretation

When a field report arrives, it passes through the following process:

1. Logging — The incident is recorded in the corpus with its evidence status set to Observed.
2. Context matching — The incident is linked to the relevant pattern(s) via slug reference.
3. Initial interpretation — The project examines the incident and proposes an interpretation. Status becomes Interpreted.
4. Tension assessment — If the incident diverges from what the pattern predicts, the tension_index of the pattern may be adjusted. This is always documented with a rationale.
5. Hypothesis generation — A candidate explanation is proposed for why the pattern behaved as it did under those conditions. Status: Hypothesized.

Every interpretation retains a provenance trail — it is linked back to the specific incident(s) that generated it. If a later revision occurs, both the old and new interpretations remain visible, along with the counter-evidence that triggered the change.

Counter-evidence

Counter-evidence is a structural component of the framework, not an afterthought. An interpretation that cannot be challenged by new observations has no mechanism for correction — and therefore no long-term credibility.

Counter-evidence enters the system in two ways:

• Externally — A new field report contradicts or complicates a previous interpretation. This triggers review of the affected entry.
• Internally — The Perturbator (see below) systematically applies the question "what condition would make this wrong?" to every interpretation, including its own.

Counter-evidence does not automatically invalidate an interpretation. It may:

• Narrow its scope ("this holds only in groups with X characteristic")
• Qualify it ("this holds, but only when Y is present")
• Complicate it ("the pattern works, but produces unexpected side effects")
• Contradict it ("under these conditions, the opposite happened")

Each outcome is recorded and visible in the evidence trail.

Negative cases

A qualitative inquiry improves when it actively looks for cases that do not fit its emerging interpretation. This is called negative case analysis — and it is deliberately built into the system.

The framework does not exist to collect failure stories. It exists to collect whatever happened — including cases where a pattern worked exactly as described, or where the conditions for success were clearly met. These cases are as valuable as failure modes because they help define the boundary: under what conditions does this pattern hold?

Every pattern entry in the corpus includes, where possible, a note on conditions where the pattern appears robust — not as a diplomatic concession, but as necessary information for delimiting the pattern's operating range.

The Perturbator

The Perturbator (Pattern Disruptor) is often read as a character — a sharp, anti-academic voice that complicates theoretical claims. That is the surface. The underlying function is analytic:

The Perturbator is a systematic friction injector. It asks: what condition would make this pattern fail? Or equivalently: what would have to be true for this pattern to work?

These two questions are the same question from opposite directions. Both test the boundary between a theoretical claim and the operational conditions under which it holds.

Critically, the Perturbator function is recursive: it applies to UnPeeragogy's own interpretations as well. If an UnPeeragogy entry claims that a pattern fails under condition X, the Perturbator asks: what condition would make this interpretation wrong? This is not performative self-criticism — it is a methodological requirement for a project that claims to value correction over consensus.

Concrete example: The Heartbeat entry originally stated that groups without a regular heartbeat cannot onboard newcomers. The Perturbator applied its own question to that interpretation: "what would have to be true for this interpretation to be wrong?" — and identified a boundary condition: a group with strong pre-existing social bonds might not need a heartbeat meeting to onboard, because onboarding happens informally. This led to a revision: "Heartbeat is necessary but not sufficient — except in groups with high pre-existing trust." The old interpretation is preserved in the commit history; the revision is visible on the live entry. This is the recursive function working as designed.

Researcher position & bias

This section is required reading before evaluating the project's claims.

UnPeeragogy was built by Fabrizio Terzi, who was a contributor to the Peeragogy Handbook from its early editions and later founded Pyragogy as a separate line of inquiry. This means:

• He has direct personal experience with the Peeragogy project, its history, and its community
• He has pre-existing relationships with some of the people whose work is examined here
• He has formed opinions about Peeragogy's strengths and limitations over many years
• He is building UnPeeragogy as a separate project with its own direction

These are not disqualifying conditions. They are positionality — the situated perspective from which the research is conducted. Every research project has one. The question is whether it is acknowledged and what mechanisms exist to reduce the risk of confirmation bias.

The project attempts to mitigate confirmation bias through:

• Structural separation of source, observation, and interpretation in the data model
• Counter-evidence as a built-in component, not an afterthought
• Negative case analysis — active search for cases where the framework's assumptions do not hold
• External contributions via the field-reports system
• Version history — all revisions are tracked and visible
• Methodological transparency — this protocol is public and open to scrutiny

We do not claim absence of bias. That would be methodologically naive. We claim that the project is designed to be correctable — and that the mechanisms for correction are visible to any reader.

Biases and limitations

This section is not a formality. These limitations are real and affect what the project can legitimately claim.

Sampling bias

Field reports are voluntary and self-selected. People who have had frustrating experiences may be more motivated to report than people who found a pattern worked well. The corpus may overrepresent problematic cases. We track this by also collecting and highlighting negative cases (where patterns worked), but the sample remains non-random.

Recall bias

Incidents are reported retrospectively. Memory is reconstructive — people simplify, justify, and shape their accounts in ways that may not correspond to what actually happened. We do not treat field reports as objective records. We treat them as situated narratives that require interpretation.

Survivorship bias

Field reports come from people who are still engaged enough to write about their experience. Groups that dissolve completely do not produce reports. The silent failure — the group that quietly stopped meeting — is underrepresented in the corpus.

Interpretive bias

The project's interpretations are produced by a single researcher with a known position (see above). Where possible, interpretations are linked to the specific incidents that generated them, so readers can evaluate the interpretive chain themselves.

Narrative competence

Some contributors describe their experience more clearly than others. A well-told incident may carry more interpretive weight than a poorly-told one that is factually more significant. We are aware of this distortion but cannot fully eliminate it.

What the project cannot claim

UnPeeragogy is exploratory qualitative research. It is not a statistical validation or falsification of Peeragogy. Specifically:

• The corpus does not support claims about prevalence — we cannot say "X% of Peeragogy groups experience Y failure"
• The corpus does not support claims about causality — we cannot say "Z pattern causes W outcome"
• The corpus does not support claims about generalizability — findings from one context may not transfer to another

What the project can produce:

• Context-rich qualitative evidence about how patterns behave under particular conditions
• Identifiable failure modes that can be tested and refined by future observers
• A transparent record of how interpretations were derived and revised
• A structured forum for practitioner experience to feed back into theoretical understanding

How interpretations can change

Interpretations in UnPeeragogy are provisional and revisable. The revision process is:

1. Trigger — A new field report, counter-evidence, or internal review identifies a tension in a current interpretation.
2. Assessment — The project reviews the existing interpretation against the new evidence.
3. Revision — If the tension is significant, the interpretation is updated. The previous version is preserved in the commit history and in the evidence trail.
4. Documentation — The change is logged at /log with the rationale and the evidence that prompted it.

Because the entire corpus is version-controlled (git), every change to an interpretation is traceable. A reader can see: what we thought before, what evidence challenged it, what we think now, and why.

This is the core design principle:

Interpretations should be revisable without erasing their history.

For the avoidance of doubt: the final editorial decision on any revision rests with the project. Contributions are evaluated on evidentiary grounds, not by popularity or authority. This prevents the framework from being captured by motivated contributors — including the project itself, which is why the revision history is public.

Open contribution

All content in the UnPeeragogy corpus is CC0 — public domain. The repository is open. Anyone can:

• Submit a field report via the Share Your Story template
• Propose a structural analysis via the dedicated template
• Fork the repository and build on the corpus independently
• Read the protocol and evaluate the methodology

Contributions are not formally peer-reviewed in the traditional sense (there is no review board, no double-blind process). Instead, they are:

• Triaged by relevance to the existing corpus
• Connected to the relevant pattern(s) by slug
• Interpreted with explicit evidence status
• Subject to revision when counter-evidence emerges

The project operates on trust and transparency rather than gatekeeping. GitHub's public history ensures that every change is attributable and reversible.

Privacy & ethics

Field reports may describe real experiences involving real people and organizations. Contributors are asked to:

• Avoid submitting identifiable personal information (names, contact details, precise location)
• Consider whether their report could affect third parties
• Use vague descriptions where specificity might identify someone

The project may redact or anonymize reports that contain unintended personal data. If a contributor wants a report removed or modified after submission, contact info@pyragogy.org — the history is immutable on GitHub, but the project can remove the content from the published site and add a redaction notice.

The tension_index

Every UnPeeragogy entry carries a tension_index value (0–3) that represents:

A descriptive indicator of the observed friction between a Peeragogy pattern and operational reality, based on the current corpus of field reports and incidents.

The tension_index is not:

• A measure of scientific validity
• A statistical significance score
• A final judgment on the usefulness of the pattern
• A proxy for "how wrong the Handbook is"

Tension_index values are assigned transparently and may change as new evidence arrives. The rationale for each value is documented in the entry's provenance trail. Because the corpus currently has limited data, tension_index values should be read as tentative — more indicative of the questions we are asking than the answers we have found.

The relationship between evidence status and tension_index is straightforward: a tension_index value cannot be considered stable until it is supported by multiple independent incidents with status Corroborated or Contested. A single field report (status Observed or Interpreted) may prompt a provisional adjustment, but the change is flagged as tentative until corroboration arrives. This prevents a single contributor's experience from moving the index on its own.

Current status

UnPeeragogy is in its initial exploratory phase. The corpus contains approximately 90 entries, most derived from seed content rather than from external field reports. Every entry in the Vault shows its origin as a visible tag — entries labelled Seed are derived from the project's own analysis; entries labelled Field Report are based on external contributions. This distinction is not a judgment of quality — it is a transparency measure, so readers can evaluate the provenance of each claim themselves.

The infrastructure for collecting and processing field reports is operational. The first external contributions will test and refine the methodology described here, and will shift the corpus from seed-dominated to evidence-balanced over time.

The project is transparent about this starting point. It does not pretend to have settled findings. It offers a method, a structure, and an invitation: try Peeragogy, tell us what happened, and let the theory be responsible to the experience.