An automated quantitative content analysis process for humanitarian logistics research Open Access

Any good research project requires high-quality data. However, such data are difficult to get, especially in the area of humanitarian logistics, because humanitarian organizations do not prioritize data collection when responding to disasters. Even when they collect data, it is often incomplete and of insufficient quality (Kunz et al., 2017). Therefore, researchers must collect their own primary data, which is difficult if they do not have established contacts among humanitarian organizations. As a result, researchers often use hypothetical data to test their findings, which questions the validity and accuracy of the models they develop (Gupta et al., 2016; Kovacs and Moshtari, 2019).

An alternative to using hypothetical data or collecting primary data is to use secondary data from humanitarian organizations, such as reports, websites, or social media. Humanitarian organizations publish large amounts of reports for their donors. These reports must be transparent and publicly available, and some governments even regulate their structure and content. For example, the US Internal Revenue Service has specific annual-reporting requirements for nonprofit and charity organizations (IRS, 2018). Public repositories of rich secondary data about humanitarian crises also exist. For example, the UN Office for the Coordination of Humanitarian Affairs runs the website ReliefWeb.int (ReliefWeb, 2018), on which it publishes maps, reports and press releases about specific disasters. Most of these resources come from humanitarian organizations. The UN World Food Program operates a similar website focusing specifically on humanitarian logistics operations (Logistics Cluster, 2018). Content on social media (Facebook, Twitter, LinkedIn, etc.) is another source of rich data that organizations and their followers generate.

These few examples show that large amounts of secondary data are available, and most are potential sources of data for research in humanitarian logistics. However, the challenge is that these data are unstructured and most often in text form. Analyzing such data requires a method called content analysis, which is a “research technique for the objective, systematic, and quantitative description of the manifest content of communication” (Berelson, 1952, p. 18). Shapiro and Markoff (1997) further define it as a methodological measurement applied to text that makes it a rigorous and reproducible way of analyzing written documents. While content analysis has been used since the beginning of the 20th century, it has evolved significantly over the past few decades as digitalization of communication has increased and computer-assisted tools for text analysis have been developed (Krippendorff, 2004).

Content analysis has been used for a variety of research approaches in different disciplines. In operations management, multiple studies have used it for conducting literature reviews (see e.g. Demeter et al., 2019; Seuring and Gold, 2012; Seuring and Müller, 2008; Spens and Kovacs, 2012; Spens and Kovács, 2006). Other studies have used content analysis to analyze social media feeds related to operations management (see e.g. Chae, 2015).

Over the years, content analysis has evolved into two different approaches, qualitative and quantitative. The present paper focuses on the latter and proposes an automated quantitative content analysis (AQCA) approach using the four steps suggested by Seuring and Gold (2012). In the first step, the researcher selects the documents to be analyzed, then assesses their descriptive characteristics in the second step (e.g. year of publication, length, authors). In the third step, the researcher identifies the dimensions of interest for the study and their related categories, resulting in the creation of several codes (i.e. labels for specific concepts). Finally, the researcher assigns these codes to words or sentences in the text (“coding”), constituting the basis for identifying patterns in the documents.

Several of these steps are time consuming and rely on the researcher’s subjective judgment. For example, deriving codes from the data may require reading the documents multiple times. This paper proposes an approach to automate the development of codes. The coding process is another time-consuming task because the researcher must read the documents carefully and label words or sentences of interest with the corresponding codes. While this is feasible with a small set of documents, e.g., interview notes, it becomes a monumental task when analyzing thousands of pages of documents, e.g., academic papers or annual reports. This task also is mentally challenging because the researcher must remember all codes and their definitions, as well as decide on the appropriate code to use for each sentence of interest. This results in a process that is not only slow, but also quite subjective. This paper proposes an approach to automate the coding process through software, and it provides several recommendations for mitigating the risks inherent in this approach.

After coding the documents, the codes can be analyzed qualitatively or quantitatively (Hsieh and Shannon, 2005). In the qualitative approach, the researcher analyzes the document’s latent content by interpreting the text’s underlying meaning (Seuring and Gold, 2012). In the quantitative approach, which is this paper’s focus, the researcher analyzes the frequency of codes in the different documents to identify patterns (Berelson, 1952). However, code frequency may not be sufficient for extracting information of interest. This paper proposes a set of measures to address this limitation for different types of analyses.

This paper’s objective is to present the AQCA approach and provide researchers with a tool that they can use to analyze large amounts of text quickly and reliably. Although the method is generic and can be used in different fields, this paper contributes to humanitarian logistics research by providing application examples from this discipline and discussing specific opportunities and limitations. Specifically, the following research questions are investigated:

The next section presents a short literature review on content analysis and its application to humanitarian logistics research. Section 3 describes the four studies that use the content analysis method introduced in this paper, and Section 4 presents the AQCA approach. Section 5 discusses the approach’s limitations and how they can be mitigated, and Section 6 concludes the paper.

Content analysis has been used in research since the beginning of the 20th century (Hsieh and Shannon, 2005) and was formally recognized as a research method in 1941 (Schreier, 2012). Ten years later, Berelson (1952) published the first leading textbook on this method, which set the basis for content analysis as a scientific research methodology (Schreier, 2012). Berelson (1952) defined content analysis as a method for the systematic, objective and quantitative description of the manifest content of communication. Shortly after that, Kracauer (1952) criticized the quantitative approach suggested by Berelson, arguing that it fails to consider the complex meaning of text and the context in which it is used. He proposed a qualitative approach to content analysis. The benefits and limitations of both approaches have been the subject of a continuous debate, but both are now recognized as acceptable methods to conduct content analysis (Duriau et al., 2007; Hsieh and Shannon, 2005; Schreier, 2012; Seuring and Gold, 2012). The next subsections describe both approaches in more detail.

Quantitative content analysis is the original approach suggested by Berelson (1952), and it focuses on counting words’ usage frequencies in documents. The words are coded into categories, and their relative frequency is analyzed to create a numerical summary of the message (Neuendorf, 2016). The basic assumption is that a word’s usage frequency indicates the importance of the subject (Guthrie et al., 2004; Krippendorff, 2004). Kracauer (1952) questioned this assumption and argued that a word that occurs only once in a text can convey a strong message about the content of the text. Another limitation is the quantitative content analysis’ inability to take into account the context in which the words are used (Kracauer, 1952). Despite some criticism, the quantitative approach has been recognized widely (Neuendorf, 2016; Patton, 2002; Tashakkori, 2006) and used extensively because of its ability to evaluate the content of documents systematically and objectively (see e.g. Barringer et al., 2005; Doucet and Jehn, 1997; Kabanoff et al., 1995; Riff et al., 2014).

The widespread digitalization of written communication and the development of information technology have enabled the use of computers to support the quantitative content analysis process (Krippendorff, 2004). Stone et al. (1966) undertook the first attempts in this direction in 1966, yielding a tool that still is in use today (Harvard General Inquirer, 2018). Computer-assisted content analysis has increased the scalability of the method, thereby allowing for use with large volumes of data by automating tasks such as coding and data storage (Duriau et al., 2007).

The emergence of big data and data-analytics tools further reinforced the role of computers in quantitative content analysis on large data sets, where it is often referred to as “text mining” or “text analytics.” Whereas quantitative content analysis and text mining use similar tools, they differ at the analysis level. While content analysis concentrates on analyzing word frequencies, text mining uses Natural Language Processing tools to interpret communication and identify patterns that human readers might overlook (Neuendorf, 2016). Text mining also has a predictive ability that is not pursued in quantitative content analysis (Yu et al., 2011), rendering it suitable for analyzing social media (e.g. using posts to predict future patterns without a preconceived theoretical basis). Therefore, while text mining and quantitative content analysis have some overlap, they are not intended for the same purposes, or as Neuendorf (2016) puts it, text mining is “almost (but not quite) content analysis” (Neuendorf, 2016, p. 235).

Qualitative content analysis emerged in reaction to Berelson’s (1952) method of quantifying the content of documents. Kracauer (1952) criticized the loss of accuracy in this approach, as rich qualitative content is lost when transforming it into numbers. He also noticed that communication often contains “latent meanings” that require some form of interpretation, which only can be done through a qualitative analysis. In light of these criticisms, qualitative content analysis has been developed as a method for analyzing not only the manifest content of documents, but also the latent meaning of “what is written between the lines” (Carney, 1972, p. 25). With this approach, the researcher identifies thematic patterns that emerge from a close reading of the text (Neuendorf, 2016), while also considering the context (Schreier, 2012). This human interpretation, which requires multiple readings of the text (Hsieh and Shannon, 2005), is slow and inevitably subjective (Kracauer, 1952), making it more difficult to achieve high reliability (Schreier, 2012).

Qualitative content analysis has been applied in several studies (see e.g. Altheide and Schneider, 2013; Elo and Kyngäs, 2008; Forman and Damschroder, 2007; Mayring, 2000; Schreier, 2012). Hsieh and Shannon (2005) categorize qualitative content analysis in three categories, depending on how the codes are developed (inductively, deductively, or summatively). Mayring (2000) proposes a similar classification and differentiates between inductive category development (categories derived from the content) and deductive category application (categories derived from existing theory).

Both Berelson (1952) and Krippendorff (2004) question the distinction between quantitative and qualitative content analysis. They argue that any text is qualitative in nature, even when some characteristics are later converted into numbers. Berelson (1952) has criticized the generally small samples analyzed in qualitative approaches and the high degree of subjective interpretation required of the coder.

This subsection presents a number of applications of content analysis in the field of humanitarian logistics research. So far, this method has been used mainly for conducting literature reviews and analyzing social media content. Caunhye et al. (2012) used qualitative content analysis to identify the type of optimization models used in emergency logistics literature. Kunz and Reiner (2012), Leiras et al. (2014) and Seifert et al. (2018) used qualitative and quantitative content analysis to categorize papers along different dimensions. Content analysis also has been used to analyze social media content in the context of humanitarian operations. Ashktorab et al. (2014) used content analysis to extract information related to disaster response from Twitter posts. Chew and Eysenbach (2010) analyzed the content of Twitter posts to monitor the outbreak of the swine flu pandemic in 2009. Mejia et al. (2019) used content analysis to examine how updates posted on a crowdfunding platform affected the amount of donations collected. Yoo et al. (2016) used content analysis to study the diffusion speed of information on Twitter in 2012 during hurricane Sandy. Content analysis can also be used to analyze reports and news, which Kovacs and Moshtari (2019) suggest as a relevant way for researchers to better understand real problems of humanitarian organizations.

Several risks exist when using content analysis in the context of humanitarian logistics research. Trustworthiness of the data source is sometimes questionable, particularly when using big data generated on social media platforms (Kovacs and Moshtari, 2019; Wells and McMillan, 2017).

Another risk when using social media content is the potential bias due to the “echo chamber” effect, in which users are “are exposed only to information from like-minded individual” (Bakshy et al., 2015, p. 1130). Such situations ultimately reinforce individuals’ pre-established opinions and perspectives (Colleoni et al., 2014) and lead to a viral propagation of some posts. As a result, the number of posts on social media is not necessarily a representative assessment of the population’s opinion, especially when users express opinions on issues such as donations, humanitarian aid policy, or government actions in the aftermath of a disaster.

Another limitation of using content analysis of social media data comes from the “digital divide” (Norris, 2001), which excludes the population without internet access from having a social media presence. Although this divide is shrinking worldwide as internet access becomes more widely available, a large share of the population in developing countries still does not have web access and, thus, no presence on social media. Therefore, any study using social media, especially in the context of a developing country, will be biased toward the population that has an online presence, and non-connected people will be missed (Crawford, 2013). This may lead researchers to wrong conclusions (Lazer et al., 2014). This limitation is reinforced in the event of a disaster because the most affected population probably would not have access to social media (due to power or communication outages), or would not be spending time on social media because they would be focusing on disaster survival and/or recovery (Madianou, 2015). Thus, any results from a content analysis using social media are only representative of the least-affected people who post on social media.

Trustworthiness of data is also an issue when analyzing other forms of written communication. While academic papers and reports from reputable international organizations can generally be considered trustworthy, this assumption is questionable with other forms of materials. In today’s world of increasing politicization of humanitarian crises and information warfare, there is a serious risk that published documents or even newspapers contain false information. Host governments have a strong influence on humanitarian operations in some countries (Dube et al., 2016; Kunz and Reiner, 2016), and may force humanitarian organizations to omit information in their reports.

This literature review provides a limited overview of content analysis history and its application to humanitarian logistics research. Several books present this methodology and its use in multiple disciplines (see e.g. Berelson, 1952; Carney, 1972; Krippendorff, 2004; Neuendorf, 2016; Schreier, 2012), and several papers describe how to conduct content analysis in logistics and supply chain management research, (see e.g. Seuring and Gold, 2012; Spens and Kovács, 2006). This paper does not intend to replicate these studies, but rather to extend them to the field of humanitarian logistics research. In particular, it proposes an Automated Quantitative Content Analysis (AQCA) approach to guide researchers in humanitarian logistics through the content analysis process.

This section briefly presents four papers that applied the AQCA approach, and specifically describes the content analysis process.

Kunz and Reiner (2012) used the AQCA approach to conduct a literature review of 174 papers on humanitarian logistics. This study categorized the papers along six dimensions. The first five dimensions came from existing theory and previous reviews (e.g. type of disaster, research methodology). The sixth dimension categorized the most prevalent situational factors described in each paper (socio-economic, environmental, infrastructure and government). The study’s objective was to identify situational factors underrepresented in extant research and suggest areas in need of more research.

We used an inductive approach to identify 120 codes related to situational factors (Mayring, 2000). Three researchers then independently assigned these codes to the four categories of situational factors to ensure validity and reliability (Duriau et al., 2007). We used the software Atlas.ti to auto-code the papers with the 120 codes (9,233 words coded), then analyzed the occurrence of codes from each category. The relative frequency of codes allowed us to identify the most represented situational factors in each paper.

Seifert et al. (2018) was another literature review using the AQCA approach. This study identified and analyzed all papers at the nexus of humanitarian supply chain management and refugee assistance. It used the AQCA approach to categorize papers according to three dimensions: the paper’s main focus (supply chain management vs refugee assistance), research approach (quantitative vs qualitative) and the type of operation described in the paper (disaster relief vs development aid). Because only two categories existed in each dimension, the objective was to determine whether the paper has more codes in one or the other category.

Gold et al. (2017) is not directly related to humanitarian logistics, but employs the AQCA approach in a way that could be used for humanitarian logistics research. Our hypothesis was that agrifood companies focus most of their communication on the sustainability of their actions, although this does not always translate into real action at the operational level. We selected the four largest agrifood companies and analyzed their activity and sustainability reports using the AQCA approach. We inductively derived the codes from the documents and assigned them to four performance objectives: sustainability, quality, cost and responsiveness. The reports’ auto-coding with 52 codes yielded 5,143 words coded.

In this working paper (Dube et al., n.d.), we use the AQCA approach to analyze annual reports of four humanitarian organizations over five years (representing over 4,600 pages of content) to find how insecurity affects them over time. We inductively derived 35 codes from the documents, and auto-coding yielded 6,873 words coded. The frequency of insecurity-related codes in each report shows an increase in insecurity in humanitarian operations from 2010 to 2015. However, comparing different organizations showed that insecurity does not affect all organizations equally.

The approach presented in this section builds on the experience gained from the four papers described in the previous section. This paper does not claim to develop a new method of content analysis. Instead, its contribution is to present an approach that has proven itself valuable in four papers, then share some best practices on how to apply it. The approach follows the four steps proposed by Seuring and Gold (2012), which are inspired by Mayring (2000) and Krippendorff (2004). Although the AQCA approach uses the software package Atlas.ti, other tools for text analysis exist and could be equally used (e.g. NVivo, RapidMiner).

Figure 1 shows the terminology used throughout this paper. At the first level is the dimension to be analyzed. For example, in the case of a literature review, this could be the “method of the paper.” Each dimension then has two or more categories, e.g., “Quantitative” and “Qualitative.” Each category has one or more corresponding codes (which are developed inductively from the documents or deductively from theory). Such codes could be “Interview,” “Case study,” “Delphi,” etc. The codes are the basis for the auto-coding process, in which the software tool assigns the codes to the actual words in the document. For example, the software would assign the code “Interview” to all words with the same root, such as “interviewer,” “interviewee,” “interview” and “interviews.”

Before starting the material collection, one must define the study’s purpose based on the research questions to be investigated. The study either could aim to categorize documents along different dimensions (e.g. a literature review, as in Papers 1 and 2), or to identify specific constructs within documents to find patterns (e.g. analysis of reports, as in Papers 3 and 4). The study’s purpose then allows for defining the unit of analysis, which could be academic papers or reports, news articles, or social media content.

The next step is to identify the source for collecting the documents and define the search criteria and any filters applied for excluding particular documents. It is extremely important to define and delimit the material carefully (Seuring and Gold, 2012) because adding documents later in the process is particularly challenging. Indeed, the inductive development of codes must be done based on all documents in the selection, or on a carefully selected sample of documents. Adding documents once the codes have been developed is a potential source of bias.

The last step in the material collection is reporting the sources of documents (e.g. databases, websites) and describing the search criteria and filters used to include or exclude documents in the selection. This ensures transparency and allows researchers to replicate the study in the future.

The objective of this step is to present the documents’ characteristics to be analyzed (Seuring and Gold, 2012). In the case of a literature review, one typically would present the number of papers published per year, number of papers per journal, number of papers per author, etc. The descriptive analysis can show interesting trends concerning the evolution of the body of literature. For an analysis of reports, a descriptive analysis shows the number of documents analyzed, publication years, number of pages in each document, etc.

In this step, the researcher defines the dimensions to be analyzed, as well as their related categories. When the purpose of the study is to categorize papers, or categorize documents’ content, at least two mutually exclusive categories should be made for each dimension (Krippendorff, 2004), so that each document (or content source) can be assigned to one of the categories. It is important that these dimensions and categories are clearly defined (Guthrie et al., 2004) and connected to theory to ensure the findings’ internal validity (Spens and Kovács, 2006). Common dimensions and their related categories in the field of humanitarian logistics are:

• context of operations (disaster relief or development aid) (Kovács and Spens, 2007);

• cause of disaster (natural or man-made) (Van Wassenhove, 2006);

• speed of disaster’s onset (slow or sudden) (Van Wassenhove, 2006);

• phase of disaster (mitigation, preparation, response, or recovery) (Kovács and Spens, 2007); and

• type of research methodology (various categories, see e.g. Altay and Green, 2006, Caunhye et al., 2012).

A literature review usually also adds one or more specific dimensions in line with the review’s purpose. In such a case, the dimension and its categories might not exist in extant literature, but nevertheless should be grounded in theory. When analyzing secondary documents, e.g., activity reports, dimensions also should be based on existing theory and aligned with the research questions.

Once the researcher defines the dimensions and related categories, the next step is to identify the codes for each category. This can be done in two ways: deductively (based on theory) or inductively (based on the documents’ content). In the deductive approach, the researcher derives codes from existing theory (Mayring, 2000), or from a content analysis dictionary, e.g., the Harvard General Inquirer (HGI) (Stone et al., 1966). Such dictionaries provide exhaustive lists of words related to specific concepts, but only a limited number of dimensions are available in these dictionaries (HGI currently covers only 26 dimensions, and none of them is specific to humanitarian logistics research, obviously). This deductive approach is particularly suitable when the study aims to extend existing theory, and codes can be extracted from literature (Hsieh and Shannon, 2005). It has the benefit of increasing construct validity because it relies on codes that have been used before and validated.

When existing theory is limited, and no predefined codes are available from extant research – as often is the case in humanitarian logistics research – one must take an inductive approach and derive the codes from the documents to be analyzed (Mayring, 2000). All four papers presented in Section 3 followed this approach. The benefit of the inductive approach is that one gets the information directly from the documents, without being influenced by preconceived codes or theoretical perspectives (Hsieh and Shannon, 2005). However, deriving codes from documents is a long process that requires reading the material repeatedly. The AQCA approach automates the inductive development of codes with a software tool, as described in Figure 2.

In Step 1, the software tool creates a count of all words in all documents, which results in a (very long) table of words with their frequency counts. In Paper 1, this list contained 40,000 words. In Step 2, all words occurring less than x times are eliminated to focus only on the most relevant words. x is a threshold defined by the researcher, depending on the level of detail required and the number of documents in the selection. For example, Paper 1 used a threshold of five (resulting in 12,000 words used five times or more), and Paper 4 used a threshold of 10 (resulting in 8,300 words used ten times or more). In Step 3, the researcher “scans” through the list of remaining words to identify the ones related to the dimension to be analyzed. This process is daunting at first, but after a while, it becomes manageable and surprisingly fast (200 words per minute is easily feasible). Sorting the words into alphabetical order helps in this process because it allows for identifying and grouping words with the same roots together (unless they have different prefixes). To increase reliability and ensure that no word is missed, one or two other researchers can do the same process independently. Alternatively, the same researcher can repeat the process multiple times to increase the likelihood of identifying all words of interest.

In Step 4, the researcher combines the selected words that have the same roots. For example, say the word “politics” and all its variations (“political,” “politically,” “geopolitical,” “politician,” etc.) are combined into a word, “politic.” It is important to document these word combinations because they will be needed for the auto-coding process. In Step 5, the researcher assigns the words to codes. Depending on the desired level of granularity in the analysis, each word is assigned to one code (e.g. the word “politic” and its variations are assigned to code “POLITIC”), or multiple words are assigned to one code (e.g. the words “school,” “schooling,” “educator,” “education,” “teach,” “teacher” and “teaching” are assigned to code “EDUCATION”).

In Step 6, the researcher assigns the codes to the different categories. It is recommended to have one or two other researchers do this same task independently to ensure the coding’s reliability, then to verify inter-coder reliability with a measure such as Cohen’s κ, Fleiss’ κ (Landis and Koch, 1977), or Krippendorff’s α (Krippendorff, 2004). Such measures work in most cases, but they have limitations in terms of number of coders or number of categories that they can cover. An alternative measure is the percentage agreement between coders (Kovács et al., 2012), which measures what percentage of codes were assigned to the same categories.

Once the researchers assign the codes to categories and settle any discrepancies, one can produce the final list of codes for each category. The paper always should provide this list to increase transparency and allow other researchers to replicate the study or use the identified codes for future research.

Once the codes have been defined, it is time to start the coding work, i.e., to label the words or sentences in the documents with the relevant codes. This can be done manually with software such as Atlas.ti or NVivio, but this task is cumbersome because the researcher must remember all codes and their related words while reading through the material. Therefore, manual coding is limited to relatively small selections of short documents. Also, the manual coding of text is subject to errors (e.g. overlooking a word that needs to be coded, or coding a word with the wrong code). The AQCA approach addresses these limitations by using the software’s auto-coding function.

Before starting the auto-coding function, the researcher enters all codes in the software using a specific syntax. In Atlas.ti, the researcher defines a list of all words to be coded with a particular code. In doing so, it is important to include all words with the same root, as identified in Step 4 of the inductive development of codes. In Atlas.ti, the sign “*” is the operator for including all variations of a word, and the sign “|” is the Boolean OR operator. The following example shows how to define the code “POLITIC” for all instances of the words “politics,” “political,” “politically,” “geopolitical” and “politician.” Note that the “*” at the beginning of the word “politic” ensures that the word “geopolitical” is included:

The next example shows how to define the code “EDUCATION” for coding all instances of the words “school,” “schooling,” “educator,” “education,” “teach,” “teacher” and “teaching”:

After entering all codes into the software, the researcher selects the documents to be coded and starts the process. At the end of this process, Atlas.ti shows the documents with the coded words highlighted in the text, and the assigned code indicated in the margin.

Post-coding verification is an important follow-up step to ensure reliability in the coding process. It is a manual verification of the coding work, which intends to confirm that the coded words have the intended meaning. The researcher manually removes the coding from the words that have a different meaning because of context. For example, in Paper 3, the code “QUALITY” was used for identifying words related to food quality. The post-coding verification showed that the word “quality” had been coded in the term “high-quality debt,” which obviously is a financial term that is unrelated to food quality. Thus, all codes assigned to this term were removed.

Post-coding verification also shows words coded in sections of the document that should not be analyzed. For example, the researcher might find a word coded in the running titles of reports, which falsely influences subsequent analysis (because the word appears on each page of the report). One also might need to remove all codes assigned in the reference section of academic papers because they could bias the analysis. In extreme cases, the researcher might even need to delete a code entirely if it has an ambiguous meaning that could be assigned to two different categories. For example, in Paper 3, the code “NATURE” had to be deleted because it was sometimes used to describe food quality, while other times, it was used in the context of the company’s sustainability practices.

Post-coding verification is a time-consuming task, but is nevertheless important to ensure the study’s reliability. In any case, this verification step still is much faster than reading all the content of all documents and assigning codes manually.

The coding output is a code-document matrix that indicates how many times each code appears in each document (absolute frequency). Summing up the occurrence of all codes in a category makes it possible to find the frequency of words from each category in a document. While this absolute measure provides some useful insights, it is difficult to compare the frequency of words across different categories, or between documents of different lengths. The AQCA approach uses different measures of relative frequency that address these limitations and allow for normalizing absolute frequency, thereby comparing frequencies across categories and across documents.

This measure is particularly useful when allocating documents to different categories, e.g., for a literature review. The challenge is that codes from one category might be used more often than others in all documents, and it would not make sense to allocate all documents to this category. For example, in Paper 1, words from the category “socio-economic” situational factors were the most frequent in all documents. However, some documents obviously focused more on this situational factor than others. The measure presented in Equation (1) allows for identifying the category that is over-represented in a document by comparing category frequency in that document with the average frequency of that category in all documents:

Equation (1) defines the relative frequency of codes from category i in document j (F_i,j) as the absolute number of code occurrences from category i in document j (x_i,j), divided by the average occurrence of codes from category i in all documents $(X¯i)$⁠. Equation (2) shows the calculation of $X¯i$⁠, in which N is the number of documents in the selection. If a category appears more often in a document than in the average of all documents, its relative frequency is larger than 1, meaning that this category is over-represented in that document. Once the relative frequency of codes from all categories in a document is known, one can assign the document to the category with the highest relative frequency, as shown in the following equation:

When using the method in Paper 1, the papers with no over-represented category were removed from the analysis (i.e. the maximal F_i,j<1) because no category is mentioned more in that paper than the average of all papers. Papers in which the difference between the two highest relative frequencies was smaller than 0.1 also were removed because the distinction between the two highest frequencies is unclear. Doing so ensured that the categorization included only papers with a clearly over-represented category. We used this measure in Paper 1 to identify the situational factor most represented in each reviewed paper. Based on this measure, we found that socio-economic and governmental situational factors were understudied.

This measure is useful when allocating documents to two categories used similarly often, i.e., there is no one category that always is used more often, like in the previous measure. Paper 2 uses this measure to categorize papers based on their main focus, either “supply chain management” or “refugee assistance.” The following equations show the measure for calculating the relative frequency of codes from both categories:

Equation 4 defines the relative frequency of codes from category 1 in document j (F_1,j) as the absolute number of occurrences of codes from category 1 in document j (x_1,j), divided by the sum of the absolute number of occurrences of codes 1 and 2 in document j. Equation (5) presents the same formula for F_2,j. Because only two categories exist, the sum of both relative frequencies is equal to 1. Once the relative frequency of both categories in a document is known, one can assign the document to the category with the highest relative frequency, as shown in the following equation:

Running this analysis in Paper 2 showed that some papers have a similar relative frequency of codes from both categories. This indicates an unclear distinction between both categories, and the method cannot make a reliable judgment. One option would be to remove these documents from the selection; another option is to create a middle category, called “Balanced,” which contains all documents with an unclear distinction (Equation 7). For example, Paper 2 used limits of 0.33 and 0.66 for the middle category of the dimension “main focus of the paper,” but the researcher must decide what limits to use, depending on the number of documents in this category and the study’s purpose:

In Paper 2, only 6 percent of the papers were assigned to the category “Balanced.” This was a surprising finding and indicated that the body of literature is focused heavily either on supply chain management or refugee assistance, with only few papers combining both disciplines.

This measure is useful when comparing the frequency of codes from a category across documents with various lengths. Paper 4 uses this measure to compare the frequency of insecurity-related words in activity reports over time and across different organizations. One organization, the International Committee of the Red Cross publishes activity reports that can be over 600 pages in length, whereas the activity reports from another organization, World Vision International, are only 70 pages long. Therefore, comparing the absolute frequency of codes would not be useful. The measure in Equation (8) solves this by normalizing the frequency to the number of words in the document:

Equation (8) defines the relative frequency of codes from category i in document j (F_i,j) as the absolute number of occurrences of codes from category i in document j (x_i,j), divided by the number of words in paper j (W_j). Because the result of this division is a small number that is difficult to comprehend, it is multiplied by 1 million to express the frequency as the number of codes per million words. In Paper 4, we used this measure to monitor the evolution of insecurity-related issues over the years, and found that humanitarian organizations have been increasingly affected by insecurity. Comparing this measure across organizations also shows that insecurity affects the organizations in their operations differently.

This section discusses the limitations of using the AQCA approach and how to mitigate them. The method, like any quantitative content analysis approach, is a simplification of written communication. It tries to transform rich content into numbers, which has the benefit of enabling faster and more reliable coding, but comes at the expense of losing valuable qualitative content from the text. This limitation can be addressed by clearly specifying the objective of the method (analyzing quantifiable aspects of content) and acknowledging the limits of the approach.

Another limitation is the method’s inability to consider the context in which a word is used, which is particularly important when doing research in humanitarian logistics. This limitation can be addressed by conducting a thorough post-coding verification to identify instances where a word is not used in its intended meaning and remove them. The risk of missing important contextual information can also be addressed by combining the ACQA approach with a qualitative content analysis approach, as we did in Papers 1 and 2 (but which is not the focus of the present paper). In Paper 4, we combined the ACQA approach with a case study method to capture the rich contextual knowledge of practitioners.

There is a risk of selection bias when identifying words related to the dimension of interest from documents (inductively deriving the codes from the documents). This can be mitigated by having multiple researchers do the same task individually, then comparing their selections of words and assessing the degree of inter-coder reliability. Another risk of selection bias occurs when assigning codes to categories. This must be addressed by having multiple researchers assigning codes separately, then assessing the degree of inter-coder reliability.

There is also a risk of bias caused by coded words appearing in specific sections of a document. If a word appears in the running title (i.e. on each page of a document), it will be over-represented in that document and probably should be removed from the occurrence count. Systematically removing the coding from these sections is not always required, but the researcher must assess this risk and decide accordingly whether to make such changes.

If the selected documents are intended for different target audiences and fulfill different purposes, they might be difficult to compare because they use different wording, and an auto-coding approach might not work well. For example, comparing meeting minutes across different humanitarian organizations would be acceptable, but it is not advisable to compare newsletters, meeting minutes and activity reports in the same content analysis (unless one can ensure that the same wording is used for the studied dimension). This must be carefully considered during the material collection process.

In addition to these limitations in the AQCA approach, several risks exist when using content analysis in the context of humanitarian logistics research, as presented in Section 2. Researchers need to be extremely cautious when analyzing content of news media, and should discuss potential political affiliation of the sources of content and describe how to mitigate resulting biases. Content of humanitarian organization reports must be used with great caution when the government of the host country imposes severe restrictions on humanitarian organizations, such as limitations on public communications.

Using content analysis to study social media content also raises the issue of trustworthiness of data. While the researcher can never guarantee a total veracity of the content, it is nevertheless important to discuss possible biases caused by false information and describe measures taken to mitigate this risk. The digital divide is another limitation when doing content analysis of social media data. The content posted online is only representative of the population with access to the internet, not necessarily the most affected population. Humanitarian logistics researchers using content analysis on social media data should be extremely cautious when interpreting their results, and clearly discuss how the digital divide impacts their results.

The echo chamber effect, in which social media users follow only posts of like-minded individuals, may lead to certain posts propagating virally and skewing the result of the quantitative analysis. Researchers using the AQCA approach have to be aware of this risk and discuss how to mitigate it.

This paper presents an AQCA approach that has been applied successfully in four papers. This approach is particularly suitable for humanitarian logistics research because good data are very limited and difficult to collect (Kovacs and Moshtari, 2019; Kunz et al., 2017). However, it is possible to overcome this challenge by using documents published by humanitarian organizations, which must communicate transparently with their donors; thus, plenty of documents are available. Public repositories of such documents are a reliable source of information about particular disasters (see e.g. Logistics Cluster, 2018; ReliefWeb, 2018). However, analyzing large quantities of documents reliably is challenging because it is extremely time consuming and involves subjective judgments.

This paper demonstrates how to analyze such documents using an automated approach, which saves significant amounts of time and provides a more objective content assessment. In particular, the paper’s first contribution is a process for identifying words of interest from documents, then assigning them to different categories. This inductive development of codes ensures that the content analysis captures all concepts of interest mentioned in the documents. The paper’s second contribution is a process for automatically coding documents with a software tool, allowing for coding of large documents quickly and reliably (one paper analyzed 4,600 pages of content with the AQCA approach). The third contribution is a description of three measures of relative code frequency, which researchers can use to categorize documents, identify common themes in documents, or track the use of certain themes over time. Finally, the paper discusses several limitations and risks that researchers must address when applying content analysis in the context of humanitarian logistics research.

The AQCA approach presented in this paper will help researchers analyze the content of large sets of documents and use them as a convenient data source. Considering the recommendations and limitations presented in this paper will enable them to conduct rigorous and reliable research on secondary documents in humanitarian logistics.

This paper presents the automated quantitative content analysis (AQCA) approach that the author has developed and applied in three published papers (Kunz and Reiner, 2012; Gold et al., 2017; Seifert et al., 2018) and in a working paper. Although the author was responsible for developing the AQCA approach and implementing it in these papers, the author has benefited greatly from suggestions and input from co-authors, and the author is grateful for their contributions (Nonhlanhla Dube, Stefan Gold, Gerald Reiner, Lysann Seifert, Taco van der Vaart and Luk Van Wassenhove).