Discovering Utilization Patterns in an Online K–12 Teacher Professional Development Platform: Clustering and Data Visualization Methods

With increased access to the Internet, teachers have access to multiple opportunities for teacher professional development (PD) online. Carter (2004) identified four characteristics of a successful online teacher professional development: 24/7 access to online resources; community of learners; community of practice; and professional support. Each of these characteristics is designed to support the immediate PD needs of teachers through online self-paced resources.

The EdHub Library is an online PD platform for K–12 teachers and school administrators maintained by the College of Education at the University of Missouri–Columbia. The EdHub Library is part of the Network of Educator Effectiveness (NEE). The Bill and Melinda Gates Foundation initially funded the EdHub Library in 2014 in partnership with the University of Missouri–Columbia. NEE is a comprehensive educator evaluation system that tracks multiple measures of educator effectiveness, including classroom observation of teachers, units of instruction, professional development plans, and student surveys (Network for Educator Effectiveness, n.d.).

Similar large online PD platforms exist, such as PE Central and MyTeachingPartner. PE Central is an online professional development platform for physical education teachers. With 162,000 visitors, PE Central provides lesson plans, best practices, classroom management, use of technology, and assessment ideas (Hanson, Pennington, Prusak, & Wilkinson, 2017). MyTeachingPartner is another example of online self-paced resources where teachers have access to three types of resources that aim to improve teacher-student interactions: a video library of best practices of teacher-student interactions, a college course, and individualized web-mediated coaching sessions (MyTeachingPartner, 2017).

The purpose of the EdHub Library is to support K–12 teachers, principals, university teaching assistants, and teacher prep students in the advancement of best practices in teaching and learning in PK–12 and higher education levels (EdHub Library, n.d.). School districts with a subscription to NEE have access to a video library of best practices in classroom teaching, a video library of examples for scoring classroom observations, a catalog of self-paced online modules, copyrighted assessment instruments, and yearly principal calibration training.

In the EdHub Library, teachers can browse online resources from a list of topics and search content organized by teacher standards. EdHub can be accessed directly on the homepage at www.theedhub.org or through the NEE online teacher evaluation tool that tracks multiple measures of teacher effectiveness. Online resources are presented in modules with several activities that prompt teachers to reflect on their teaching practices or generate new ideas on instructional strategies for classroom implementation using the journal feature. Teachers have control over sharing journal entries with school administrators and reviewing their activity history from the user dashboard. In addition to the site features available to teachers, school administrators have administrative permissions that allow them to review teacher activity and control access within the school group.

While we know about the purpose of the website and how many users are registered on EdHub, little is known about what users do once they are on the website. The purpose of this study is to discover the utilization patterns of content and site features using a clustering method and data visualization from 460,721 records available from October 2015 to February 2017.

Google Analytics (GA) data are collected in the form of dimensions and metrics using a script embedded in the site and online resources. Dimensions are qualitative attributes of the data such as the page, type of browser, and location. Metrics are quantitative measures of data such as time spent and number of views of a page. To make sense of the data collected, combining dimensions and metrics generates insights from user interactions with online resources and platform features.

This section outlines existing literature related to the components of effective teacher professional development, evaluation of in-person and online PD, and research design for investigating utilization patterns of professional development platforms.

Multiple studies have investigated the components of teacher professional development and the impact of PD on teaching practices and teachers’ professional careers. Desimone (2011) argued that the core features of effective teacher PD lie in content focus, active learning, coherence, duration, and collective participation. These core features of effective teacher professional development focus on the alignment of subject matter content with the school district’s goals and policies by allowing teachers to provide feedback about their instructional practices and challenges during PD activities. In contrast to the effective elements of teacher PD described by Desimone (2011), Borko (2004) described teacher PD as a contextualized educational system that consists of a PD program, facilitators as the providers of PD, and teachers as recipients of PD.

Guskey (2002) argued that the ultimate goal of staff development is to address the motivational aspect of teachers engaging in professional development. A change in teachers’ beliefs and attitudes occurs when teachers participate in PD and change their classroom practices when teachers see evidence of increased student learning outcomes. While Guskey’s model considers teachers’ personal beliefs and dispositions to professional development, Bechtel and O’Sullivan (2006) described contextual aspects in schools that affect teacher change, including school culture, micropolitics of schools, level of support, and workplace conditions. With a positive school culture, high personal interactions, and high support from principals in an adequate workplace, these contextual elements encourage teachers to experiment with new methods of teaching in their classrooms.

Coldwell (2017) examined how teacher PD influences career progression and workplace retention of science teachers in British schools. In the analysis of 25 semistructured phone interviews and over 500 teacher surveys from PD sessions, teacher PD was an intermediate factor of workplace retention where teachers are “much more likely” to stay in the teaching profession. The study also identifies individual factors that influence career progression and workplace retention. These individual factors are motivation, PD engagement, school policies, and the ability to demonstrate knowledge. The findings suggest that highly engaged teachers in PD are very likely to stay in the profession because they are given opportunities to improve and validate their knowledge that makes them more motivated and confident as science teachers.

Rice and Dawley (2009) surveyed 259 teachers, administrators, and instructional coaches from U.S. K–12 virtual schools to understand professional development practices. Their findings suggest that online teacher professional development should be characterized by offering multiple opportunities for professional development, ongoing training sessions, coach or peer mentoring sessions, student-focused activities, customized pieces of training based on teacher needs, and training aligned to standards.

A similar survey of 800 teachers by the National Foundation for the Improvement of Education (1996) describes eight aspects of effective professional development for beginning and experienced teachers: allow flexible scheduling of PD activities throughout the school year, extend professional development opportunities while students are on vacation, customize PD based on student data, align teaching practices with school district and state standards, allow teachers to design their PD based on their expertise, collaborate with teachers and administrators to promote responsibility and effectiveness, expand teaching roles into other areas of responsibility in the school, and mentor new teachers with the help of experienced teachers and administrators.

Fischer et al. (2018) investigated the effects of teacher professional development participation, teacher characteristics, and contextual school characteristics on teachers’ implementation of instructional practices in advanced placement courses. In examining national datasets from 133,336 students and 7,434 teachers, the findings suggest that teachers’ PD participation influences implementation of instructional practice, experienced advanced placement teachers tend to implement new instructional practices, and economically challenged schools are likely to implement fewer instructional practices. While these findings on teacher and school characteristics are relevant, student characteristics also play important roles in teachers’ implementation of new instructional practices. For example, students with educated parents perform slightly better,and the school socioeconomic level affects students’ advanced placement performance.

While multiple studies investigated the characteristics of teacher professional development from the teacher perspective, the following literature describes evaluation studies of high quality in-person and online teacher professional development.

Lindvall, Helenius, and Widerg (2018) identified critical features of high-quality teacher professional development by analyzing how PD interventions impact student achievement of Swedish students from second to ninth grade and examining the content focus of PD materials from two large PD programs in mathematics education. In the first analysis, student achievement data were collected before and after teachers attended PD interventions. Teachers who participated in PD sessions showed significant student achievement gains and showed more positive development in their subject. In the second analysis, PD materials were analyzed by coding lessons in five categories: teachers’ and students’ knowledge, didactics, teachers’ actions, lesson design, and reflection on teachers’ learning practice. Their findings suggest that high-quality PD should exhibit a level of coherence or alignment between a teacher’s current knowledge and the proposed teacher change from the PD that is suitable for his or her current level of knowledge and skills. While the two mathematics PD programs were similar in context, the goals of these programs were to improve a teacher’s lesson design and content knowledge and allow for experimentation with lesson implementation in the classroom.

Zhang, Lundeberg, Koehler, and Eberhardt (2011) examined the affordances and challenges of video-based teacher professional development. In this PD program, 26 science teachers were asked to analyze instructional practices of three types of video recordings: a video of any teacher, a video of their own teaching, and a video from a fellow teacher. In the first type of video, participants witness someone else’s teaching practices and challenges. In the second type of video, participants watch their own teaching practices in the classroom and receive feedback from their peers. In the third type of video, participants watch a video from a fellow teacher that focused on learning new techniques from a colleague. The common challenges among the three types of videos were related to the lack of exemplary teaching videos, inadequate length of videos for extended observation of teaching practices within the classroom context, and lack of video examples for different grade levels and subjects.

Abell et al. (2007) conducted summative evaluations of nine in-person teacher professional development projects in mathematics and science education in order to understand the contextual factors of successful PD implementation in Missouri schools. Their evaluation approach was to create project profiles based on six components program background, program design, participants and their schools, quality implementation, outcomes, and recommendations. Project profiles relied on field notes from two site visits, participants’ interviews, student learning outcome data, and two surveys related to demographic information and PD satisfaction. The researchers argued that contextual factors play an essential role in designing and implementing effective teacher professional development by measuring student learning outcomes data and teacher PD satisfaction and analyzing contextual variables.

With the advent of educational technologies, teachers have multiple venues to engage in professional development opportunities online that allow them to address their immediate classroom challenges (O’Brian, 2016). The following literature identifies research in online teacher professional development with instruction that takes place 80%to 100%online, based on the Online Report Card by Allen and Seaman (2016).

Yoo (2016) investigated the effects of online professional development on teacher self-efficacy using the Teacher’s Self-Efficacy Scale developed by Tschannen-Moran and Woolfolk Hoy (2001). Teacher self-efficacy is defined as the teacher’s level of confidence about his or her ability to implement instructional choices that affect student learning. The teacher self-efficacy scale was administered to 148 participants before and after enrolling in the 5-week online program. After taking the online teacher professional development, participants reported increased self-efficacy regarding enhancement of teaching skills, attainment of professional goals, and acquisition of new pedagogical knowledge.

Frey (2009) investigated the effects of project-based online professional development experiences on the performance of K–12 students with disabilities and teachers’ instructional practices. The online professional development offered participants with 16 weeks of project-based coursework and journal reflections that allowed teachers to describe the performance of the classroom and a target student. The qualitative analysis of discussion boards and journal entries suggested that teachers improved their skills related to PD activities, students with disabilities exhibited improved classroom performance, and improved student performance affected teachers’ readiness to implement evidence-based instructional practices.

Whitaker, Kinzie, Kraft-Sayre, Mashburn and Pianta (2007) evaluated the use of an online teacher professional development called MyTeachingPartner by analyzing three data sources from 213 teachers, including web server logs, teacher evaluation surveys, and focus groups. Teachers were assigned to three groups with three modes of access to online PD materials. The first group, Materials group, had limited access to website resources using an iBook computer. The second group, Web group, had full access to the website resources with reflection tools and video demonstrations. The third group, Consultancy group, had allinclusive access similar to the Web group and video conferencing access to instructional coaches. Regardless of the mode of access, the three groups exceeded their monthly login requirements. Even though the Materials group had limited access to PD materials, this group exhibited longer site visits than the other two groups with additional site access and features. Their findings indicate that teachers will invest time in professional development if activities are deemed personally or professionally useful.

The researcher’s approach to analyzing large web analytics data is to use a clustering algorithm to derive utilization patterns of resources and site features. Several studies have used the simple k-means algorithm to understand patterns of educational digital libraries, online courses in higher education, and massive open online courses.

Khoo et al. (2008) examined the session length web metrics produced by various educational digital libraries. While web metrics provide insights into the behaviors in a site’s use and its users, challenges exist when triangulating tracking data with other sources of data and making firm inferences about users’ intentions solely based on web metrics. Web metrics measure user behavior on a website, but not the knowledge or attitudes of users. Bollen and Luce (2002) assessed the user characteristics and retrieval behaviors in a digital library by generating clusters to describe research interests and preferences from document retrieval networks. The generated retrieval patterns for digital resources allowed the examination and ranking of journal collections by individual user groups.

Valsamidis, Kontogiannis, Kazanidis, Theodosiou, and Karakos (2012) proposed a method for investigating patterns of student activity and online course quality from login data using the simple k-means clustering method. The researchers developed three metrics to assess the quality of online courses that were calculated from standard web metrics, such as sessions, page and unique pages, available in a learning management system. The online course quality metrics pertain to the enrichment of the educational material, disappointment or the number of interactions with web pages and interest or the number of uniquely accessed web pages in a course. By data mining standard web metrics and deriving new online course quality metrics, a cluster analysis was performed to obtain cluster patterns for understanding student engagement and online course usability.

Rodrigues et al. (2016) performed a cluster analysis of student engagement patterns in a massive open online course with 1,558 enrolled students. Seven patterns showed the levels of student engagement regarding activity completion, number of comments, and answers in discussion forums. Furthermore, seven patterns were grouped into three categories of student engagement. In the first category, 15.98%of students were highly engaged in activities, discussion boards, and answering questions. In the second category, 26.06% of students exhibited sporadic engagement behavior throughout the course. In the third category, 57.96%of students was the disengaged group with one third of activities completed and minimum interaction.

While existing research identifies positive effects of online teacher PD and success factors in high-quality online PD environments, only handful studies have documented user behavior in large online teacher professional development platforms. The need to analyze online behavior provides insights on how teachers use self-paced instructional modules and interactive site features in online teacher professional development environments. While web metrics are widely used in tracking user behavior and generate large amounts of data about resource utilization, the representation of web metrics data through a cluster analysis allows the observation of patterns from site features and resources when grouped based on similar characteristics.

The purpose of this study is to discover the utilization patterns of the content and site features of a K–12 online professional development platform. By data mining web metrics records from Google Analytics (GA), this study allows a comprehensive examination of the patterns between new and returning visitors by applying the simple k-means algorithm and visualizing natural groupings or cluster outputs (Jain, 2010).

The following research questions guided the design of the study: (1) What dimensions and metrics in GA are potential predictors of clustering? (2) What behaviors do new and returning visitors exhibit when using the platform? (3) What are the content and site feature patterns present from the predictor metrics and dimensions? The following section provides a detailed account of the data extraction, manipulation, computation, and deployment of results.

The research context is a large online content management platform with 50,425 K–12 teachers and school administrators seeking professional development in the state of Missouri in the United States. A total of 460,721 web metrics records from the platform are available from October 2015 to February 2017. GA collects metric and dimension properties when a tracking script snippet is assigned to online resources and platform features. The web metrics reports of 17 months are available for extraction from the free edition of GA.

This study adheres to the cross-industry standard process for data mining methodology. The process is a popular data mining methodology that brings organization into data mining projects (Shearer, 2000). This approach requires six systematic steps in any data mining project: (1) research understanding, (2) data understanding, (3) data preparation, (4) modeling, (5) evaluation, and (6) deployment. The following subsections describe each phase.

Research Understanding Phase. The EdHub Library is built on Plone, an opensource content management system, which provides a mechanism for controlling user and content access. Each school year, access tokens are provided to teachers and principals to control access to the platform. At a system level, the EdHub Library is integrated with GA to track user and content activity. In this data mining project, the analysis aims at examining the patterns among users, instructional materials, and site features.

Data Understanding. To make sense of GA records, the site and content taxonomy was required to understand the overall structure of the EdHub Library. The taxonomy is derived from the web hosting indexing structure. Additionally, the researcher is responsible for creating and deploying instructional materials in the web repository in a structured manner. The Google Analytics Dimensions and Metrics Explorer provides descriptions of dimensions and metrics extracted from the platform shown in Table 1 (“Google Developers,” n.d.).

Data Preparation Phase. After extracting 460,721 records in a tabular format, descriptive statistical measures were performed to identify missing cases. This preparation work makes it possible to create data visualizations in Tableau software to showcase patterns between new and returning visitors, the most searched terms, the most accessed resources, and the most accessed features. Tableau is data visualization software that produces interactive data visualizations (Tableau, 2017).

Modeling phase. A feature selection was performed to understand the potential predictors of site utilization among the dimensions and metrics. The simple k-means algorithm was used to develop similar groupings or patterns for online resources and platform features among the predictor variables. The simple k-means algorithm was performed by running the dataset using 2, 10, 20, 30 and 40 cluster parameters by standardizing Euclidean distances with ten folds of cross-validation and 50 iterations. Euclidean distance is a measure of similarity and dissimilarity among data points. Modeling activities were performed in the data mining workspace of TIBCO Statistica (TIBCO Statistica, n.d.).

Evaluation Phase. To validate the selection of the cluster parameter, the Davies-Bouldin validity index was used to evaluate the internal validity of clusters. The Davies-Bouldin validity index measures the average similarity between clusters. A lower validity index translates into a better cluster configuration regarding separation and compactness (Kovács, Legány, & Babos, 2005). After manipulating several cluster parameters, a cluster parameter of 20 produced the lowest validity index. Table 2 shows the index for each cluster parameter.

Deployment Phase. The deployment phase involves presenting the data mining efforts to understand how teachers and school administrators use site features and resources. The data mining results are presented in data visualizations in the form of bar charts for searched terms, resources, and features. Also, visualizations are presented in heatmaps to show cluster classifications organized by user type and page.

A total of 460,721 cases were extracted for 13 variables from October 2015 to February 2017 available in GA. Table 3 shows counts of 212,719 for new visitors and 248,002 for returning visitors. Part of the interest of this study is to understand how these dimensions and metrics cluster together using the k-means algorithm to develop utilization patterns.

While all 13 dimensions and metrics variables are important in this investigation, a feature selection was performed to obtain the potential predictors of the page variable. Based on the feature selection, the best predictors of the page variable were page depth, sessions, pageviews, time on page, day of the week, bounce rate, new users, session duration, and user type. These predictors will be used in the simple k-means clustering algorithm. However, city, browser, and users variables were not potential predictors.

Tableau Desktop was used to visualize the overall counts for the most searched terms, the most accessed resources, and the most accessed features between new and returning visitors. For the most searched terms in Figure 1, new and returning visitors used the search function to find professional development materials related to examples of units of instruction, critical thinking, and problem solving. For the most accessed resources in Figure 2, new and returning visitors accessed the library directory landing page and recertification online training. For returning visitors, the most accessed resources were content knowledge, formative assessment, and classroom observation training videos. For the most accessed features in Figure 3, new and returning visitors accessed the EdHub Library through the online teacher evaluation tool. In most cases, new visitors checked their user dashboard for content and group access and performed journal tasks. By contrast, returning visitors checked the library topics in the administrator library aimed at school administrators.

Seven out of 20 cluster patterns describe utilization patterns of resource pages and features for new and returning visitors. The remaining 13 clusters overlap with the previous patterns. It is important to note that patterns do not represent a sequence of actions that occur in the site. Instead, patterns show the number of cases in which actions occur the most frequent and paint the “picture” of how many events unfold in the site by examining prominent classifications or groupings.

Pattern 1: New Visitors—Teacher or School Administrator—User Registration Through the Online Teacher Evaluation Tool. This pattern shows new visitors, either teachers or school administrators, accessing the site through the online teacher evaluation tool to sign up for an account through the registration page on the EdHub homepage. In Figure 4, this cluster shows classified cases with the following utilization pattern: (1) accessing EdHub through the online teacher evaluation tool, (2) signing up for an account using the login form, (3) reviewing content subscriptions, (4) verifying group access, (5) accessing content list of topics, (6) viewing group membership, (7) receiving a successful password reset, (8) reviewing teacher standards descriptions, (9) accessing terms and conditions on the registration page, (10) logging into the site, (11) getting an error with the wrong group access token, (12) reviewing content subscriptions, and (13) resetting a password.

Pattern 2: Returning Visitors—Teacher or School Administrator—Registered Users Through the Online Teacher Evaluation Tool. This pattern shows returning visitors, either teachers or school administrators, accessing the site through the online teacher evaluation tool. In Figure 5, this cluster shows classified cases with the following utilization pattern: (1) accessing EdHub through the online teacher evaluation tool, (2) reviewing content subscriptions, (3) verifying group access, (4) signing up for an account using the login form, (5) viewing the group listing page, (6) logging out of the site, (7) reviewing teacher standards descriptions, (8) viewing token content access, (9) reviewing group subscriptions, (10) getting an error with the wrong access group token, (11) accessing online resources related to school administrators, (12) receiving a successful password reset, and (13) accessing online resources related to speech pathologist.

Pattern 3: New Visitors—School Administrators—Training, User Subscription, and Password Reset. This pattern shows new visitors, school administrators, accessing a recertification training available to school administrators through the EdHub site directly. The recertification training allows school administrators to practice scoring teacher interactions in the classroom. To access the recertification training and copyrighted materials, users need specific content access the EdHub Library. In Figure 6, this cluster shows classified cases with the following utilization pattern: (1) accessing recertification training, (2) accessing terms and conditions on the registration page, (3) resetting a password, (4) entering content token access, (5) reviewing teacher standards descriptions, (6) verifying group access, (7) reviewing content subscriptions, (8) accessing the library content list, (9) viewing members in a group, (10) entering the wrong password, (11) logging out of the site, (12) signing up for an account using the login form, and (13) viewing the group listing page.

Pattern 4: New Visitors—Teachers—Journaling and Library Content Related to Cognitive and Affective Engagement. This pattern shows new visitors, teachers, accessing journal entries and reviewing library content related to assessment topics, and teacher indicators related to cognitive engagement (1.2) and motivation strategies (5.1). In Figure 7, this cluster shows classified cases with the following utilization pattern: (1) viewing journal entries, (2) accessing the library content list, (3) viewing the journal landing page, (4) reviewing teacher standards descriptions, (5) reviewing content subscriptions, (6) reviewing NEE Teacher Standard 1 or content knowledge, (7) viewing the group listing page, (8) reviewing NEE Indicator 1.2 or cognitive engagement, (9) accessing assessment topics, (10) verifying group access, (11) reviewing NEE Indicator 5.1 or affective engagement, (12) signing up for an account using the login form, and (13) reviewing the about Edhub page.

Pattern 5: Returning Visitors—School Administrators—Administrative Viewing of Journal Entries. This pattern shows returning visitors, school administrators, accessing the site directly to view teachers’ journal entries and review library content related to instructional leadership topics and classroom observation videos. In Figure 8, this cluster shows classified cases with the following utilization pattern: (1) viewing journal entries, (2) reviewing teacher standards descriptions, (3) reviewing user group preferences, (4) accessing assessment topics, (5) accessing online resources related to school administrators, (6) reviewing content subscriptions, (7) verifying group access, (8) viewing the group listing page, (9) accessing online resources related to instructional leadership topics, (10) reviewing teacher standard NEE Teacher Standard 1 or content knowledge, (11) accessing online resources related to administrator classroom observation training videos, (12) accessing formative assessment topics, and (13) accessing online resources related to speech pathologist.

Pattern 6: Returning Visitors—School Administrators—Administrative Group and User Preferences. This pattern shows returning visitors, school administrators, accessing the group and user settings pages through the online teacher evaluation tool. These returning visitors have access to administrative settings to control group settings and member access. However, teachers do not have access to administrative settings. In Figure 9, this cluster shows classified cases with the following utilization pattern: (1) viewing group members, (2) reviewing user preferences, (3) accessing group preferences through the online teacher evaluation tool, (4) logging out of the site, (5) accessing content knowledge topics, (6) accessing assessment topics, (7) accessing school administrators topics, (8) accessing professional development topics, (9) accessing formative assessment connection topics, (10) managing user subscriptions, (11) accessing administrator classroom observation training videos, (12) reviewing content subscriptions, and (13) reviewing NEE Indicator 1.2 or cognitive engagement.

Pattern 7: Returning Visitors—Teachers— Group and User Preferences. This pattern shows returning visitors, teachers, accessing the teacher group and individual user settings through the online teacher evaluation tool. In this particular pattern, site pages related to administrative settings are missing since teachers who are members of a group do not have access to group controls. Only school administrators have access to administrative settings. In Figure 10, this cluster shows classified cases with the following utilization pattern: (1) reviewing user preferences, (2) accessing group preferences, (3) reviewing teacher standards descriptions, (4) reviewing content subscriptions, (5) viewing the group listing page, (6) accessing beginning teacher assistance topics, (7) accessing assessment topics for beginning teachers, (8) accessing assessment topics, (9) verifying group access, (10) accessing platform tutorials, (11) accessing assessment connection topics, (12) accessing modules in assessment connection topics, and (13) accessing professional development topics.

In this study, 460,721 web analytics records from October 2015 to February 2017 were clustered together using the k-means algorithm to develop content and site feature utilization patterns of an online K–12 teacher professional development platform. In this study, the free version of Google Analytics was used to extract 13 variables (page, user type, browser, city, page depth, day of the week, new users, users, sessions, bounce rate, session duration, pageviews, and time on page).

A feature selection was performed to discover the predictors of the page variable, which tracks the content and features of the platform. Based on the feature selection, city, browser, and users were not significant predictors of the page variable. Nine predictor variables for the page variable were found: user type, page depth, day of the week, new users, sessions, bounce rate, session duration, pageviews, and time on page. By selecting the lowest Davies-Bouldin internal validity index, a cluster parameter of 20 was performed on the dataset to form utilization patterns by user type and page. The site taxonomy of the platform was obtained from the web server to interpret clusters.

While prominent clusters do not show specific search behavior of users, exploring what users typed in search queries allows further understanding of teachers’ needs for professional development. Users searched for examples of units of instruction, examples of professional development plans, and classroom training videos.

Users’ search queries highlight the need for teachers to find practical and concrete implementation examples of teaching strategies that model the skills in their classroom context. Research shows that teachers face challenges in implementing new teaching strategies rather than merely learning the theory behind teaching strategies. Ermeling (2010) argued that teachers look for instructional solutions that lead to detectable improvements in student outcomes. Joyce and Showers (2002) stated that teachers are likely to explore training that models a new skill that is close to their classroom setting. Studies by Peery (2002), Blank and De Las Alas (2009), and Darling-Hammond et al. (2009) explored the positive outcomes between teacher practice and student learning when teachers focus on training with similar teaching skills and practices.

Seven patterns describe use cases for teachers and school administrators when seeking PD and performing administrative functions. Patterns 1 and 2 are related to new and returning visitors, either teachers or school administrators, obtaining a registration to EdHub through the online teacher evaluation tool, reviewing their group and content subscriptions, and resetting passwords.

Pattern 3 describes new visitors, school administrators, accessing the EdHub site directly for training, checking their group subscription access, and performing password resets. In this particular pattern, these new visitors are school administrators going through recertification training that requires administrative access to EdHub to verify teacher members in a school group.

Pattern 4 describes new visitors, teachers, accessing the site directly to perform journal tasks, reviewing descriptions of teacher NEE Indicators 1.2 (cognitive engagement) and 5.1 (affective engagement), and accessing topics related to assessment or NEE Standard 7. The absence of administrative tools in this pattern indicates that these new visitors are teachers.

Pattern 5 involves returning visitors, school administrators, accessing the site directly to view teachers’ journal entries, reviewing group and personal preferences, and viewing topics related to assessment, instructional leadership, speech-pathology, and classroom observation training videos.

Patterns 4 and 5 explain how teachers and school administrators behave on the site by discovering what is available in the library through the NEE Standards descriptions page or the library topic listing before performing journaling activities. Users can review descriptions of standards and their related topics from the NEE Standards page. Alternatively, users can review descriptions of PD topics through the library homepage. Regardless of how users access topics, users are presented with the alignment of a topic and standard to aid in the selection of journal activities.

Pattern 6 involves returning visitors, school administrators, reviewing their group membership and personal preferences. In this particular pattern, school administrator access EdHub via the online teacher evaluation tool to review administrator PD materials, which is only accessible to school administrators, and manage administrative settings of a group. Regarding professional development pieces, this pattern contains topics related to assessment and professional development.

Pattern 7 describes returning visitors, teachers, reviewing user preferences, group membership, and content subscriptions. Similar to pattern 6, teachers access EdHub via the online teacher evaluation tool to review teacher standard descriptions, getting started guides of the platform, and topics related to assessment, professional development, and beginning teacher. In this pattern, however, administrative pages are not present in the cluster.

Although prominent classifications are useful in understanding which site features and content occurred most frequently for the majority of users, small cluster patterns provide additional knowledge about the most utilized instructional resources and the issue of referrer spam.

Small Classifications For Sessions. In the cluster output for sessions, small classifications with a high number of sessions appear among returning visitors in topics related to instructional strategies, examples of units of instruction, and critical thinking, whereas new visitors access topics about the process of evaluating units of instruction.

Small Classifications for Page Depth. In the cluster output for page depth, small classifications with a high page depth appear among returning visitors who access group preferences, whereas new visitors access the journal tool.

Small Classifications for Pageviews. In the cluster output for pageviews, small classifications with a high number of pageviews are related to new visitors accessing video examples of indicators 1.2 (cognitive engagement) and 7.4 (effects of instruction). Returning visitors have a high number of pageviews when accessing their user preferences page, and video examples of teacher indicator 4.1 (problem-solving and critical thinking) and 1.2 (cognitive engagement).

Small Classifications for Session Duration. In the cluster output for session duration, small classifications with a high session duration can be observed in topics related to academic vocabulary for new visitors, and administrative group settings and recertification training for returning visitors.

Small Classifications for Time on Page. In the cluster output for time on page, small classifications with a high time on page suggest that new visitors spend more time on resources related to speech-language pathology, recertification training, and formative assessment, whereas returning visitors spend more time on topics related to instructional strategies and examples of units of instruction.

Small Classifications for Referrer Spam. Referrer spam is a troublesome issue in GA tracking that makes repeated website requests using fake links to the site being spammed for advertising purposes even though these spam websites never visit the target site. In GA, website administrators can turn on the bot filtering option to block web traffic from referrer spam. Even when the bot filtering option is checked in GA for this study, less than 3 percent of classifications show referrer spam websites for bounce rate, new users and session variables. The clustering algorithm was able to detect patterns of three referrer spam websites (traffic2-cash, snip.to, and adf.ly) that sent advertising links to GA. Spammers aim to create backlinks for users to access spam sites inadvertently.

While deleting spam records from web analytics records is ideal, two challenges are present in dealing with referrer spam. First, identifying referrer spam among thousands of records is difficult because these spam websites generate new subdomains or links to disguise their domains as new visitors. Second, cleaning spam website records is time-consuming and inaccurate because not all referral sites generate new traffic for spam purposes. For example, search engines and other sources, such as school districts networks and other online PD platforms, may redirect their searches to the EdHub domain.

Even if the bot filtering option is enabled in GA, it is not guaranteed to eliminate all spam website traffic. It is important to note that referrer spam websites were not able to access and alter web analytic records because the platform requires login credentials to access site features and content. Since the clustering algorithm was able to detect very small classifications of spam websites while the bot filtering option was checked, proactive measures can be taken by creating specific filters to prevent these spam website domains from sending backlinks in the future.

By comparing the prominent seven cluster patterns, it allows further exploration of differences in tasks and site access performed between teachers and school administrators. As shown in Table 4, in four patterns out of seven, teachers and school administrators access the EdHub site through the online teacher evaluation tool. In the three remaining patterns, teachers and school administrators access the EdHub site directly.

When accessing the EdHub site through the online teacher evaluation tool, new visitors who are teachers (1) register for an account and (2) verify group and content membership, whereas new visitors who are school administrators (1) verify group members, (2) check content membership, and (3) access training materials.

When accessing the EdHub site directly, new visitors who are teachers (1) access the journal tool, (2) review teacher standards descriptions, and (3) search for content related to NEE Indicators 1.2 (cognitive engagement) and 5.1 (affective engagement), whereas new visitors who are school administrators (1) verify group members and (2) check content membership.

A fundamental difference exists in the tasks for new teachers when accessing the EdHub site through a third party and directly. When accessing the site directly, new teachers tend to engage more with the site by accessing the journal feature, searching teacher standard descriptions, and seeking online PD based on teacher standards. When the site is accessed through the online teacher evaluation tool, new teachers perform the necessary tasks of registration, and checking their group and content memberships. For school administrators, either new or returning visitor, they perform administrative tasks regardless of access.

Returning visitors who are school administrators accessing the EdHub site, either through the online teacher tool or directly, show more engagement with reading teachers’ journal entries, checking group preferences, and accessing various online PD topics. Returning visitors who are teachers, accessing the EdHub site through the online teacher tool, review assessment and beginning teacher topics. Unfortunately, returning teachers do not engage with the journal tool in the future.

Several online professional development topics are present across the patterns pertaining to cognitive engagement, formative assessment, beginning teacher, affective engagement, administrator library, speechlanguage pathologist, getting started with EdHub, classroom observation video examples, and examples of units of instruction. However, the following topics are not present: communication, data analysis, Common Core State Standards and Next Generation Science Standards, and family and community involvement. All topics are available to teachers and administrators, except for the administrator library. Descriptions of these topics present in the patterns are shown in Table 5.

The results of this study can be used to understand user behaviors of teachers and school administrators, prioritize the development of professional development topics, and improve the navigation of the EdHub Library. Users will benefit from having quick access to the most utilized professional development materials when resources are placed on the homepage. As the Network of Educator Effectiveness grows, stakeholders will need to address the increased access to online professional development through the teacher evaluation tool and develop online professional development materials that target specific teaching skills with several classroom contexts.

Teachers and school administrators perform the essential tasks based on their role. Teachers obtain a bird’s-eye view of what is available in the EdHub Library before completing journal activities. Even though cluster patterns do not show returning teachers accessing the journal tool for future activities, cluster patterns show returning school administrators reading teachers’ journal entries and performing administrative functions.

While Google Analytics provided a large dataset for developing utilization patterns, the same dataset can be used to investigate the relationships among online professional development materials. A hierarchical clustering analysis can be performed for different topics to explore further how new and returning visitors navigate from one online resource to another. A network analysis can also be performed to understand the connections among online resources by metric or dimension. By investigating teachers’ journal entries, a text mining activity will provide users’ sentiment and word patterns of teachers’ professional challenges, classroom practices, and instructional implementation challenges.