A measurement framework of crowd intelligence

With the breakthrough and continuous development of new-generation information technologies such as the Internet of Things, cloud computing, big data, artificial intelligence (AI) and so on, the future society is becoming an open, ecological, large-scale, self-organized, intelligent and interconnected system called the “crowd cyber” (Chai et al., 2017; Qiu et al., 2018). The crowd cyber is a convergent system consisting of physical space, information space and consciousness space. As the interactions among these components conform to different laws of motion, the behavioral outcomes of the crowd cyber exhibit the following characteristics: unity of stability and variability, unity of order and disorder, unity of determinism and randomness, unity of heterogeneous organization and self-organization, unity of knowable and unknowable and unity of controllable and uncontrollable, etc. These changes will have a disruptive effect on the future structure and form of industry, leading to the individualization of products, the decentralization of production, the miniaturization of government and the publicization of the means of production.

Many philosophers and brain scientists around the world have studied intelligence and its nature since time immemorial, but there is still no standardized concept of what intelligence is. In general, intelligence may be high or low, and collective intelligence is higher than individual intelligence, but there is a lack of research results on how to measure it quantitatively. In the crowd cyber, people, machines, and objects are considered to be intelligent, and the originality of the crowd cyber system is that it has the intelligence of multiple groups. The purpose of the research on intelligence measurement under the crowd cyber is to assess the intelligence of individuals and groups and to support the assessment of the innovation potential of individuals and groups. This paper is based on a review of the existing literature to give a new interpretation of crowd intelligence in the form of a set under the crowd cyber and to develop an intelligence measurement model that measures intelligence based on two dimensions: intelligence quotient (IQ) and effectiveness. Four representative forms of intelligence are summarized based on the interaction of intelligent individuals in the information space. Figure 1 shows that the crowd intelligence is the product of the integration of physical space, information space and intelligent space [¹]. Intelligent space represents IQ, and there is heterogeneity in the IQ of people, machines and objects. The physical space represents effectiveness. Information space connects intelligent space and physical space, which mainly represents the relationship between intelligence and effectiveness, where there are various interactive relationships, including people and people, things and things, people and things, people and machines and so on. Here, social networks are not just about socializing people, but a “big social” network of people, machines and things. It is worth noting that all the discussions on intelligence in this paper are also based on Chai et al.'s research framework on crowd cyber.

This paper is organized as follows: We conceptualize the intelligence in Section 2. Then we introduce the four stages of intelligence evolution in Section 3. We sort out the measurement dilemma of intelligence and propose ideas for change in Section 4. Finally, we propose a measurement framework for crowd intelligence in Section 5 and conclude the research in Section 6.

We have compiled several representative categories of definitions of intelligence. The Encyclopedia Britannica defines it as “the ability to adapt to one's environment effectively”. The cognitive psychologist Gardner (1983) identified intelligence as the ability to solve problems or create products, and that these problems and products are valued in one or more cultural background. In addition, a team of 52 leading psychologists summarized the dominant view of the definition of intelligence as follows (Gottfredson, 1997). Intelligence is a very general psychological ability that includes, among other factors, the ability to reason, plan, solve problems, think abstractly, understand complex ideas, learn quickly and learn from experience. It is more than just the performance of book learning, narrowly defined academic skills, and intelligence testing. Specifically, intelligence reflects a broader and deeper understanding ability of our surroundings, the ability to “know” or “understand” what we should do.

The ancient Chinese tradition defines intelligence as a general term for both intelligence and ability, and ancient Chinese thinkers generally regarded wisdom and ability as two relatively independent concepts. In the Xunzi–Zhengming chapter, it is mentioned: “Therefore, someone who knows is said to know, and someone who knows some things together is said to be wise. Therefore, it is said that those who are able are able, and those who are able together are able. The term “wisdom” refers to certain psychological characteristics of cognitive activities, while “ability” refers to certain psychological characteristics of practical activities. The Confucian school defines wisdom and ability separately, which provides a good idea for change to establish a measurement framework for the intelligence in this paper.

Based on the above review of the definition of intelligence, we find that there are two aspects of intelligence definitions. One is professional intelligence, which is the ability to effectively achieve a specific goal in a given environment. This is equivalent to the definition given by the Encyclopedia Britannica and by Gardner. The other is general intelligence, which refers to the ability to effectively achieve a variety of different goals in different environments, which is similar to the definition given by 52 psychologists, and which is measured by IQ testing, and tends to be more widely used than just assessing someone's ability to effectively perform some specific task. The definition of general intelligence requires an intelligent actor to be not only good at a particular task, but also good at learning how to perform a variety of different tasks. To put it simply, this definition is roughly the same as “diversity” or “adaptability”, and from this perspective, “crowd intelligence” in this paper focuses more on the analysis of the intelligent performance of a collective under crowd cyber, hence general intelligence is being emphasized more.

The research most relevant to this paper is a series of studies on collective intelligence. MIT's Center for Collective Intelligence (CCI) has done a number of experimental studies to test collective intelligence in recent years, contributed to explore how people and computers can be more collectively connected, thereby producing higher-order intelligence. Regarding the definition of collective intelligence, Malone et al. (2009) argue that intelligence is something that can be produced not only in the brains of individuals, but also in groups, and that collective intelligence is considered to be groups of individuals acting collectively in ways that seem intelligent.

Based on this definition, we find that collective intelligence has actually been around for a long time, with subjects such as families, armies and corporations acting together in seemingly intelligent ways. These different forms of collective intelligence have been studied extensively by scholars in many fields of economics, sociology and psychology. Woolley et al. (2010) define collective intelligence as the general ability of a group to perform a wide variety of tasks. This description attributes the group's intelligence to a factor called “Factor C”, and this factor can improve the performance of an entire team. In IQ tests of a single individual, it is often possible to predict an individual's ability to perform a task from a single statistical factor, so this measurement taken up to the group level should be equally applicable. In an experimental study of 699 people in a group of two to five, Woolley et al. (2010) found a composite factor that determines collective intelligence, a factor that explains the group's performance on various tasks. The study showed that “Factor C” was not closely related to the average or maximum individual IQ of group members, but was related to the perceived social connectedness of group members, the balance of their participation in the task, and the proportion of females. Further, the higher the team members' perception of social relations (which can be thought of here as individual emotional quotient), the more balanced their participation (i.e. the absence of minority dominance in the discussion), and the higher the proportion of women in the team, the higher the team score and the better the performance [²]. However, at this stage of the research, it is difficult to determine what the high correlation between “factor C” and task performance is based on, and what seems to be most important is the ability of the members to cooperate and coordinate their work with each other.

The widespread use of computers and the internet has allowed the study of collective intelligence to be carried out in larger groups of people, and a new type of collective intelligence has emerged, where people and computers connect to each other and do intelligent things together (Malone and Bernstein, 2015). For example, Google collects the knowledge generated by millions of people by creating and linking web pages, and then uses that knowledge to answer questions in a very intelligent way. In Wikipedia, people around the world co-create a large and high-quality knowledge product via the internet, and all volunteer to be part of it. Greenstein and Zhu (2018) compared the accuracy of Wikipedia knowledge produced by emerging online community groups to the Encyclopedia Britannica written by experts in traditional methods, and provided insights for the Managing group-based knowledge production. Almaatouq et al. (2020) consider the influence of social networks in collective intelligence, and through network crowdsourcing experiments and computer simulations, find that adaptive social networks can facilitate the production of collective intelligence and allow the intelligence of everyone in the group to be enhanced.

Through the above literature review, we find that the exploration of collective intelligence has gradually shifted from the initial study of human group performance to the study of human-machine interconnection to produce higher-order intelligence, which is a new research opportunity brought by the internet era. However, in the era of crowd cyber, the intelligent connection of people, machines and things enables peer-to-peer trusted transactions and distributed collaboration, and the integration of intelligence will be further deepened, the definition of intelligence should be innovative, so that we propose the concept of crowd intelligence based on the concept of collective intelligence.

Li et al. (2017) was the first to propose the concept of crowd intelligence, which is in the context of the AI field and AI 2.0, arguing that the internet-based cyber-physical world has profoundly changed the information environment for the development of AI, pushing AI research into a new era of AI 2.0. Li et al. (2017) give a definition of crowd intelligence: a group of people under a certain internet-based organizational structure are motivated to perform challenging computational tasks. The above definition shows that Li et al. believe that crowd intelligence is essentially a kind of internet-based collective intelligence, and that online interactive communities and crowdsourcing activities can be considered as a kind of crowd intelligence. Under the crowd intelligence network, we believe that people, machines, and objects [³] are all intelligent, not just humans collectively solving a problem based on the internet, and thus the definition of crowd intelligence should be broader.

In the research framework of the crowd science, we believe that intelligence is universal, that is, humans, animals, individuals, groups, and machines all have intelligence, and that intelligence universality is a prerequisite for the research and development of crowd intelligence. In addition, the intelligence of human beings, machines and objects can evolve to produce the phenomenon of congregational evolution, and there are horizontal differences in intelligence between different individuals, thus crowd intelligence is measurable.

“Crowd intelligence” is a generalization of “natural intelligence”, “artificial intelligence” and “integrated intelligence”, which can be expressed as a ternary set, as shown in equation (1).

Where:

CI = Crowd Intelligence;

NI = Natural Intelligence;

AI = Artificial Intelligence; and

I I = Integrated Intelligence.

Here, for the generalization of the definition, we give natural intelligence, which is the intelligence of living organisms in nature, both human and animal, but in crowd intelligence, we focus on Human Intelligence. “Artificial intelligence”, which we adopt here as defined in the field of computer science, is considered to be the ability of a system to correctly interpret external data, to learn from that data, and to achieve specific goals and tasks through flexible adaptation (Poole et al., 1998; Kaplan and Haenlein, 2019). The term “Integrated intelligence” refers to intelligence that is integrated through the coordination of “natural intelligence” and “artificial intelligence”, as shown in equation (2).

Where:

HI = Human Intelligence;

MI = Machine Intelligence. e.g. intelligence of computers, controllers and robots; and

I I = Integrated Intelligence. e.g. intelligent CAD systems for human-machine harmony, intelligent driving systems for human-machine coordination.

Overall, crowd intelligence studies the “integrated intelligence” that combines “human intelligence” and “artificial intelligence”, and the group “crowd intelligence” that emerges from the coordination of “individual (including human, machine and object) intelligence”.

Effectiveness is a complex concept, so that some scholars describe it as a construct instead of a concept. Drucker (1963) argues that effectiveness refers to the ability to choose the right goal and make it happen, i.e. the ability to do the right thing. Denison (1990) views effectiveness as the excellence of the collective in all its aspects, especially its output. For a collective, more emphasis is placed on organizational effectiveness. Organizational effectiveness refers to the extent to which an organization achieves its objectives, which is reflected in four main areas: competence, efficiency, quality and performance. Cameron (2015) argues that organizational effectiveness is a concept based on the values and preferences of the evaluator and therefore, there is no single and correct definition of organizational effectiveness. There has been literature on the classification of effectiveness definitions, including the ideal type and bureaucratic model, the goal model, the natural system model, the strategic supporter model, etc., each of these models has different criteria for evaluating effectiveness, with the goal model being the most widely accepted and oriented toward task completion goals. In this paper, we argue that the performance of a collective intelligence is closely related to final task completion, and thus effectiveness is also a measurement dimension of crowd intelligence.

In this section, we innovatively propose several forms of low to high intelligence evolution to support the measurement of intelligence.

Stage1: The evolution of intelligence in a single individual. It is believed that intelligence includes “innate intelligence”, which is the study of natural evolutionary mechanisms based on genetics and variation, and “acquired intelligence”, which is the study of evolutionary theories and methods based on learning and training.

Stage2: The evolution of intelligence arising within the same group. Such as a team of individuals cooperating with each other, and the intelligence derived from the interaction of different individuals. This is mainly referred to as “collective intelligence”.

Stage3: The evolution of intelligence resulting from the fusion of different groups. For example, the “integrated intelligence” of the human-machine system is formed by the coordination of human intelligence and machine intelligence.

Stage4: The final form of intelligence that results from the chaotic fusion of all intelligences in crowd cyber is at the highest level of intelligence, and here we name it “Super minds”, referring to the work of Woolley et al. (2018).

First, we need to explain the concept of a measurement, which is a homomorphic mapping from an objective object to a digital object. The homomorphic mapping includes all relational and structural mappings. Searching with the keyword “measurement”, it is found that most of the existing researches are focused on software engineering field, using software metrics metadata, predicting software defects based on statistical and machine learning methods, and using metrics to improve software processes. Abreu and Carapuca (1994), Bansiya and Davis (2015), Lee et al. (2016) 

Based on the application of measurement concepts in computer science, we believe that “intelligence measurement” is based on the “IQ test” theory and method of “human intelligence” and the “Turing test” theory and method of “artificial intelligence”, to further study the relativity and measurability of intelligence, and to develop “crowd intelligence” that is applicable to multiple levels, types and aspects, so as to provide scientific measurement theory and quantitative testing methods.

The previous methods of measuring intelligence mainly include the IQ test for humans and the Turing test for machines. Turning (1950) put forward the concept of “machine thinking”, which suggests that a test subject and a test subject, while isolated from each other and unable to communicate directly, are able to communicate with each other through information transmission. After a series of questions and answers, if the test subject is unable to determine whether the person is a human or a computer system based on the information obtained, then the system can be considered to have an intellectual capacity equivalent to that of a human, which means that the computer is capable of thinking. This is the Turing Test.

Currently, there are the following main dilemmas with the measurement of intelligence. First, IQ scales on people cannot be used for testing machines and systems. Second, the Turing test does not explicitly distinguish intelligence characteristics for testing and it cannot perform intelligence level testing, i.e. it does not classify the intelligence of a machine or system. Therefore, we need to conduct new, specialized research to move the research and application of intelligence measurement forward based on previous work.

Liu (2009) proposed a generalized intelligence evaluation theoretical system including “two criteria and three rules”, and proposed a generalized operator model [⁴]. By examining their research, we can provide feasible ideas for the measurement of “crowd intelligence”.

Liu's generalized intelligence evaluation system, based on the relevant theories and methods of intelligence testing, highlights the differentiation of intelligence characteristics and the measurement of intelligence level, and further includes the investigation of the application of AI techniques and methods, so as to propose a generalized IQ algorithm. However, we find that it can only measure the intelligence level of the same type of individuals or systems, and does not measure the final performance of groups, so we need to propose a more widely used measurement framework for “crowd intelligence”.

By drawing on the first part of the Confucian definition of intelligence, we argue that intelligence should be measured in two dimensions, namely, the IQ dimension and the effectiveness dimension, with the IQ dimension being the generalized IQ operator model that we can use and improve on, and the effectiveness dimension being the performance of a group in completing a task, which in turn implies relationships between different individuals. It is worth further clarifying that the two dimensions are not independent juxtapositions, but rather progressive relationships. First, we measure the heterogeneous IQ of different individuals using the crowd IQ algorithm. Second, it is more important to recognize the relationships between different individuals, as the ability of different intelligences to cooperate with each other has a greater impact on the final performance, and we further define this interaction in terms of the “big social” network of people, machines, and things in the information space. In this section, we will give a more detailed discussion.

SORT ranking of the same intelligence type. The SORT ordering rule allows a comprehensive, multidimensional comparison of IQ levels of the same type of intelligence, and the steps are as follows.

Step1: Set $Vij$ to represent the evaluation value of the j-th intelligence characteristic index of the i-th individual, and we can obtain the intelligence evaluation matrix $(Vij)n×m$⁠, and normalize it, as shown in equation (3) [⁵]:

Step2: Normalization by Step1 leads to the final intelligence score matrix V, as shown in equation (4):

Step3: Select the optimal vector $V→+$ and the worst vector $V→-$ of the intelligent score matrix V. Set $Di+$ as the distance between the i-th individual and the optimal vector $V→+$ and $Di-$ as the distance between the i-th individual and the worst vector $V→-$⁠, and $ωi→$ as the weight vector of the intelligent characteristic index. as shown in equation (5):

Step4: Compute $DIi$ as the proximity of the i-th individual to the best intelligent individual, as shown in equation (6):

where a higher value of $DIi$ indicates a higher IQ level for the i-th individual.

According to Woolley et al. (2010), the collective intelligence of the group is not necessarily high for members with high IQ, and the ability of the individual members to cooperate and coordinate their work with each other often determines the performance of their collective intelligence. Therefore, we further argue that the effectiveness demonstrated by group cooperation can also be a dimension to measure the crowd intelligence, and to facilitate the discussion here, we give a diagram of the relationship patterns as shown in Figure 2.

Under the crowd intelligence network, intelligences interact in the information space, and even if individuals with the same IQ can have different effectiveness performance due to different relationship patterns. In the Figure 2, we use blue individuals to represent humans, green individuals to represent machines, and red individuals to represent objects. Different individuals will have different IQs calculated by the crowd IQ algorithm, with a plus sign in a circle representing an above average IQ and a minus sign representing a below average IQ. Individuals with different IQs can have different relationships, such as cooperative, competitive, antagonistic and so on. In a simplified schematic diagram, we give two types of relationships: the “+” represents a cooperative relationship, where the different intelligences promote each other, so that the relationship in the first quadrant has the highest level of crowd intelligence; the “−” represents an adversarial relationship. Intelligent individuals struggle against each other, so the relational pattern located in quadrant three has the lowest level of crowd intelligence. For the relational patterns in quadrants two and four, it is difficult to give an intuitive picture of the final level of intelligence, depending on the specific task performance, but it is worth acknowledging that both IQ and effectiveness are important to consider in this intelligence measurement framework.

Crowd intelligence is some stage between mob and collective intelligence, and collective intelligence should be a retrograde form of crowd intelligence but refinement form. In this paper, we propose a measurement framework for crowd intelligence based on the crowd cyber. First, we clarify the related concepts of intelligence and give a new definition of crowd intelligence in the form of a set, which is more general and broader than the existing definitions and is in line with the direction of further development of crowd science. Second, we innovatively propose four stages of the evolution of intelligence from low to high, and sort out the dilemma of intelligence measurement at the present stage. Third, we propose a framework for measuring crowd intelligence based on two dimensions, IQ and efficacy drawing on Confucianism. The generalized IQ operator model is optimized, and a new IQ algorithm is proposed. On the basis of this IQ measurement, four representative forms of intelligence are summarized based on the interaction of intelligent individuals in the information space. Finally, we can use more rigorous mathematical symbols to represent the logical relationships between different individuals, and consider applying the measurement framework to a real-life situation or experiment to enrich the research on crowd intelligence in the further study.

The authors thank the researchers in China Center for Internet Economy Research for their useful comments and our two anonymous referees for their helpful advice. All remaining errors are ours. They are grateful for the financial support from the National Key Research and Development Program of China (2017YFB1400100), the National Natural Science Foundation of China (NSFC71804206), the Beijing Social Science Foundation Research Base Project (17JDYJB018), the Young Teachers Development Fund of Central University of Finance and Economics (QJJ1826), the Program for First-class Discipline Construction in Central University of Finance and Economics, and the Beijing Program for “Double First-class” Construction Project of Central Universities.