How do you choose a research method?
The work is not done once the research question is clear; that is where the real bottleneck starts. This guide turns method selection from a matter of preference into a decision run through two filters: the kind of knowledge the question needs, and the structure of the data on hand.
How do you choose a research method?
A research method is not chosen on its own; the research question and the data structure choose it. The four steps are listed in order below.
Reversing this order is a common cause of getting stuck: when a researcher starts from a method they already know, the question shrinks toward whatever that method can answer. What comes out is an analysis that runs flawlessly on a technical level but does not answer the original question.
Each step narrows the options for the next.
- Reduce the question to a single sentence. The sentence should state who, what, in which context, and over what time span.
- Determine what kind of knowledge the question wants: prevalence, relationship, cause and effect, meaning, ranking, or the accumulation of existing evidence.
- Describe the data: scale type, number of observations, timing of measurement, and whether the observations are independent of each other.
- List the designs that pass both constraints, pick one from among them with a validity-based justification, and keep the eliminated ones together with the reason.
The second half of the fourth step is skipped most often, though the list of eliminated alternatives is the first thing a reviewer asks about; keeping it at the moment of decision is far cheaper than reconstructing it months later.
The choice is not made in a vacuum either; which designs the same question has already been studied with comes out of the literature review. Cooper (1988) classifies reviews along the axes of focus, goal, and coverage; Webster and Watson (2002) recommend organizing a review concept by concept, not article by article.
What is the difference between quantitative, qualitative, and mixed methods?
Quantitative research tests relationships and magnitude through numerical measurement, qualitative research describes meaning and process through text and observation, and mixed methods combines the two in a single design. The difference is not really in the type of data but in the type of question: one asks how much, the other asks how and in what sense (Creswell and Creswell, 2018).
- Quantitative design: measurable variables, a predefined hypothesis, statistical inference. Categories are known before the analysis.
- Qualitative design: interviews, observation, documents. Categories emerge from the data itself during analysis.
- Mixed design: two kinds of evidence combined in a single study. If the qualitative phase prepares the quantitative phase the sequence is exploratory; if it explains the quantitative finding the sequence is explanatory (Creswell and Creswell, 2018).
Mixed methods is not two analyses placed back to back: if the point and the rule at which the two phases merge is not written down in advance, the study becomes two separate studies published in the same article, not a single design. The point of integration is described by name in the method section: which finding feeds which phase, and which piece of evidence takes priority when they conflict.
How does the research question determine the method?
The pattern of the question determines the design; the table below shows six patterns and the design family each points to. Finding the pattern narrows the set of options considerably.
| Question pattern | What it wants to learn | Suitable design family |
|---|---|---|
| How much, how many | Prevalence, distribution, magnitude | Descriptive quantitative: cross-sectional survey, secondary data analysis |
| Why, due to which factor | Relationship, effect, cause and effect | Explanatory quantitative: experiment, quasi-experiment, multivariate model |
| How, what does it mean | Process, experience, meaning | Qualitative: interview, case study, document analysis |
| Which is better | Ranking alternatives against multiple criteria | Multi-criteria decision making (Hwang and Yoon, 1981; Saaty, 1980) |
| What is known | Accumulation of existing evidence | Systematic review and meta-analysis (Page et al., 2021) |
| What do the numbers say, what do people say | Reading two kinds of evidence together | Mixed methods (Creswell and Creswell, 2018) |
The table estimates the match, it does not make the decision; more than one design can remain standing under the same question pattern. At that point elimination is no longer done by the question but by the data constraints.
Which methods does the data structure eliminate?
Data structure is the second filter that eliminates options; the table below shows how five properties of the data narrow the choice. If the observations are not independent of each other, methods that assume independence are eliminated from the outset.
| Property of the data | How it narrows the choice |
|---|---|
| Scale type | Summation and averaging are undefined for nominal and ordinal measurement; methods based on the mean cannot be used on this kind of data. |
| Number of observations | If the number of parameters to be estimated is large relative to the number of observations, the model cannot be fitted. A small sample eliminates designs with many parameters. |
| Cross-sectional or longitudinal | Cross-sectional data does not capture change over time; questions that require measuring change need a longitudinal design. |
| Dependent observations | Repeated measures from the same person, or observations coming from the same class or institution, are not independent; this structure requires a multilevel or repeated-measures design. |
| Vague, linguistic judgment | If the input is a verbal degree rather than a precise number, frameworks based on fuzzy sets, which allow partial membership, come into play (Zadeh, 1965). |
This filter needs to be run before the data is collected; if the collected data is in the wrong scale type there is no way back, and the only thing left at the analysis stage is to shrink the question. Knowing which analysis will run while designing the data collection form is a different task from later trying to fit the method to the data.
What do validity and reliability mean?
Reliability is a measurement giving a consistent result when repeated under the same conditions; validity is a measurement actually measuring what it is meant to measure. A measurement can be reliable and still invalid (it can consistently give the same wrong value); when different instruments measuring the same construct converge, that is convergent validity, and when different constructs diverge, that is discriminant validity.
Campbell and Fiske (1959) proposed the multitrait-multimethod matrix to test these two kinds of validity together: the same construct is measured with different methods, different constructs are measured with the same method, and the resulting pattern of correlations is examined. The matrix grounds the validity claim in a pattern rather than in a single coefficient.
In method selection these concepts are not decoration, they are elimination criteria: if both designs fit the question and the data supports both, the choice is made by looking at which one closes off more validity threats.
- Internal validity: the observed relationship genuinely comes from the factor under study, not from some other explanation.
- External validity: the finding can be carried beyond the sample and the context.
- Construct validity: the measurement instrument represents the concept. Convergent and discriminant validity fall under this heading.
- Reliability: the consistency of a repeated measurement; it is a precondition for validity but does not substitute for it.
How is a method choice justified in front of a reviewer?
The justification has three parts: why the chosen method fits the question, how the data structure meets that method's assumptions, and why the evaluated and eliminated alternatives were eliminated. The third part is missing from most texts; if the decisions are on record, the justification is not written afterward, it is shown.
- State the link between the question and the design in one sentence: the question wants this, this design produces it.
- Write out the method's assumptions one by one and show how each is met in the data.
- List the evaluated alternatives and the reasons for eliminating them; the reason is often a data constraint, and writing that down is not a weakness.
- Give the validity and reliability evidence for the measurement instruments with its source.
- Write the analysis steps, the versions used, and the parameters in enough detail for someone else to repeat them.
The real criterion for a method section is reproducibility: another researcher taking the same data and the same steps should be able to reach the same result (Peng, 2011). The precondition for this is that the data be findable, accessible, interoperable, and reusable (Wilkinson et al., 2016). Provenance models that record which output came from which input and which process make this trail portable on the machine side as well (W3C, 2013).
In some designs the form of the justification is also tied to a standard: in a systematic review, how the search, screening, and inclusion steps are reported is defined item by item (Page et al., 2021), and for health-field reviews the methodological framework is additionally given in the Cochrane handbook (Higgins et al.). Where such a standard exists, the method section is written to its items, not as free text.
If an AI-assisted component was used in the analysis, the same traceability is expected of it: how the data set used was collected and what it represents (Gebru et al., 2021), and what the model was designed for and what its limits are (Mitchell et al., 2019), should be documented.
What are the most common mistakes when writing up a method choice?
The most common mistake is choosing the method before the question is clear, then rewriting the question to fit it. Six common mistakes are listed below.
- The method's name is mistaken for a justification. The name tells the reader what was done, not why it was done.
- The eliminated alternatives are not written down. The reader assumes the single option was taken without ever being weighed.
- The assumptions are not checked. The results table looks correct, but the condition it rests on was never tested.
- Mixed methods is claimed and the rule for merging is never written down. Two separate studies sit in the same text.
- The sample and inclusion criteria are left vague. Who was included, who was excluded, and by which rule; without these three the study cannot be repeated.
- The name of the software is substituted for the method. The name of a package is not the justification for the method run with that package.
What these mistakes have in common is that all of them surface at the writing stage, not at the moment of decision: method selection is spread over weeks, but the method section is usually written in the last week, by which point why a given alternative was eliminated is forgotten. Recording decisions with their justification at the moment they are made turns the writing-stage task from recollection into transcription.
Which tools carry out these steps?
The workflow above also runs on pen and paper; it gets harder when the number of methods grows and the justification for a decision is asked about a month later. SciMind's products target these two points: finding a method with its source, and keeping track of a decision that has already been made.
Related guides
References
- Campbell, D. T. and Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56(2), 81-105.
- Cooper, H. M. (1988). Organizing knowledge syntheses: a taxonomy of literature reviews. Knowledge in Society, 1(1), 104-126.
- Creswell, J. W. and Creswell, J. D. (2018). Research Design: Qualitative, Quantitative, and Mixed Methods Approaches (5th ed.). SAGE.
- Gebru, T. et al. (2021). Datasheets for datasets. Communications of the ACM, 64(12), 86-92.
- Higgins, J. P. T. et al. (Ed.). Cochrane Handbook for Systematic Reviews of Interventions. Cochrane.
- Hwang, C. L. and Yoon, K. (1981). Multiple Attribute Decision Making: Methods and Applications. Springer.
- Mitchell, M. et al. (2019). Model cards for model reporting. FAT* 2019, 220-229.
- Page, M. J. et al. (2021). The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ, 372, n71.
- Peng, R. D. (2011). Reproducible research in computational science. Science, 334(6060), 1226-1227.
- Saaty, T. L. (1980). The Analytic Hierarchy Process. McGraw-Hill.
- W3C (2013). PROV-DM: The PROV Data Model. W3C Recommendation.
- Webster, J. and Watson, R. T. (2002). Analyzing the past to prepare for the future: writing a literature review. MIS Quarterly, 26(2), xiii-xxiii.
- Wilkinson, M. D. et al. (2016). The FAIR Guiding Principles for scientific data management and stewardship. Scientific Data, 3, 160018.
- Zadeh, L. A. (1965). Fuzzy sets. Information and Control, 8(3), 338-353.
Make your method decision on the record.
Platforms built to find a method with its source, place alternatives side by side, and keep track of the decision that was made. Write to us for an institutional license or a pilot.