Using the Project Management Body of Knowledge (PMBoK) Framework When Selecting a Learning Management System

Emporia State University (ESU) has been using its current learning management system (LMS) for approximately 14 years. The software is locally hosted in the data center where a team provides support for the database, application servers, user administration, directory authentication, data feeds from the Student Information System, and enduser support.

The decision was made by the Learning Technologies team at ESU to investigate alternative systems. Much of the interest in changing systems was derived from data drawn from 2 years of internal technology surveys among students who indicated growing discontent with the current system. Students responded that they are increasingly utilizing mobile devices to access course materials, grades, and digital assets, and prefer a system that can better support their needs. They also indicated that they desire a product that offers rich social media features and better support for self-produced digital assets that can culminate in electronic portfolios, project-based learning assignments, and authentic assessments. It was determined that the current LMS lacks:

1. a contemporary look-and-feel, and ease-of-use features;

2. convenient mobile device access required of contemporary systems;

3. course notifications via social and mobile media;

4. a rich array of social media features and native access to popular social media sites; and

5. integration with digital assets, such as student-produced video and multimedia.

The back office staff expressed that the current LMS lacks:

1. a process or resource to manage/merge/unmerge multiple course sections;

2. a process or resource to manage/upload/modify student avatars/photo rosters;

3. a mechanism to streamline product upgrades and bug fixes;

4. a native mechanism to reconstitute deleted courses; and

5. the ability to customize without significant programming expertise.

Faculty and instructors expressed that the current LMS:

1. is difficult to use—too many “clicks” to accomplish tasks;

2. does not provide rich analytics that help them measure student competencies;

3. does not provide a mechanism to direct grades to the Student Information System;

4. must be brought offline at critical periods of the academic year to perform upgrades; and

5. lacks a reliable plagiarism detection system.

As a result of this feedback, a Learning Management System Task Force was empaneled with the charge of identifying a replacement system. To address this challenge, the LMS Task Force project manager employed the PMBoK (Project Management Body of Knowledge) methodology. PMBoK is often used by organizations that are either analyzing software applications or that have adopted a specific product and are in the process of integrating that product into the enterprise. It blends the principles of quality management with cost management, resource management, personnel management, and other factors to ensure a successful product launch.

Increasingly, universities and colleges are engaging project management methodologies to control complex technology changes within their enterprise. In order to effectively control and manage this level of complexity, “projects” are created that involve specialized personnel and stakeholders from across the enterprise. According to the Project Management Institute (PMI), a project is “a temporary group activity designed to produce a unique product, service or result” (PMI, 2012). Change has become a regular and expected part of most academic technical environments. Enterprise software applications are added, removed, or upgraded routinely. Traditionally prepared mid-management is often ill-equipped to control for this type of rapid software evolution. Crawford and Blackburn (1996) state,

Typical enterprise software applications, such as an LMS, are becoming increasingly complex and involve a multiplicity of stakeholders from across the institution. Some of those touch points where the LMS intersects include server administrators, database administrators, network personnel, helpdesk agents, trainers, software developers and programmers, instructional designers, distance education support personnel, video support personnel, academic administrators, faculty, staff, and most importantly, students. A typical LMS integrates with the Student Information System; the directory authentication system; the e-mail and alert notification system; an early warning system (used to identify at-risk students); a plagiarism detection system; an evaluation and analytics system; an electronic portfolio system; a web-conferencing application; a lecture capture application; a streaming video system; social media applications, and mobile applications indicate the array of associated technologies. A change or upgrade in the LMS will often mean that each of these secondary systems will be affected as well.

Software projects of this magnitude can no longer be isolated to individual departments with limited scope and limited impact to the institution. That is, they are no longer considered vertical products that only affect a limited number of users within a confined ecosystem, but rather horizontal products that cut across multiple ecosystems. Therefore, project management is the most effective mechanism to control software applications with complex change management challenges. An applicable project management framework is the previously noted PMBoK, or Project Management Body of Knowledge. The PMBoK framework is a subset of PMI. It is considered a collection of processes and knowledge areas that are accepted as “best practices” within the field of project management (Haughey, 2012). While PMBoK can be used for a variety of applications, it is particularly well-suited for enterprise software management. This is because

PMBoK is recognized as a sequential project management method. It is an ideal methodology for large and potentially dispersed interdependent teams; it is well suited to handle complex inter-dependencies between tasks and teams; it is excellent for fixed-fee projects; and it includes predefined sequence cycles that control project timelines. Whereas, other nonlinear frameworks such as Agile project management are arguably better suited for largescale manufacturing applications, such as production plants (Thangalvaadi, 2011).

Starting in mid-summer 2012, the LMS Task Force project manager began assembling the various components that would guide the product selection process. The foundational framework for the process involved PMBoK. According to Thomas (2009) “the PMBoK Guide outlines five key process groups to aid in project delivery:

1. Initiating: Setting up the project for success by identifying the right team and scope, as well as determining the relationship between the project and its alignment with the organization's overall charter.

2. Planning: Developing the relevant resources, timelines and milestones, and mapping project delivery to business priorities (i.e., risk management, communications, quality, cost/budgeting, duration and sequencing, indentifying external dependencies).

3. Executing: Assigning the project team and distributing information to ensure the proper activities are undertaken. This process also includes ensuring quality assurance methods are in place to address change management, organizational updates, possible changes to the plan, et cetera.

4. Controlling and Monitoring: Ensuring the resulting product maps back to the original plan, and risk from uncontrolled external actions is diminished. Monitoring can have a significant impact by setting up a secure infrastructure to:

   • monitor quality, costs and schedule;

   • manage stakeholder relationships, risk and contract monitoring,

   • identify discrepancies (or variations) within the project schedule, and

   • provide the PMO more control.

5. Closing: Making sure you have delivered everything expected of the project. Once you close, you need to review the project vis-à-vis the plan and likewise ensure contract closure.”

At ESU, the LMS Task Force project manager used this framework to build a series of action items that guided the entire product review sequence:

1. Secure Executive Sponsorship: The first step in any evaluation of this magnitude is to ensure there is support from senior administration. Typically, the chief academic officer (provost) and the chief information officer must agree to sponsor the project plan. Rationale for the change, along with anticipated timeline, community involvement, expected costs, and project methodology are clearly communicated to the executive sponsors. If it is eventually deemed that an alternative product will be selected, these executives will likely be the individuals that carry this decision forward to the president, faculty senate, and the deans for formal discussion and approval.

2. Interviews With and Data Collection From Other Colleges and Universities: The second step in the process was to analyze results from other colleges and universities that had recently embarked on similar product evaluations. Interviews were conducted with the project managers from those campuses to determine how they approached the product evaluation cycle. This proved to be extremely useful information. In each case, the project manager shared how he or she was able to collect information and resources that provided further insight as to how they approached the evaluation cycle. In some cases, the campuses not only engaged project management methodology, but also were able to marshal such resources as human factors experts to conduct usability testing and analysis prior to product selection.

3. Conduct a Literature Review: The third step in the process was to conduct a thorough literature review from other campuses who had documented their analysis. In many cases, the campuses posted their analysis and results on publicly accessible web sites. Data were collected from the University of Texas, North Carolina State University, the University of California-Chico, University of San Francisco, and Pittsburg State University (Kansas) to name a few. The LMS Task Force Project Manager analyzed their data and evaluation methods to generate an approach for ESU. In addition, EDUCAUSE provided helpful documentation regarding the LMS review and selection process (EDUCAUSE, 2010).

4. Empanel the LMS Task Force: In order to ensure a broad representation of constituents across campus had equal input into the decision process, a Learning Management System Task Force consisting of 23 individuals was empaneled. Included were faculty representatives from each of the four academic schools and colleges; the office of distance education; student services; learning technologies; information technology back office support personnel (database, web administration, Student Information System integration, e-mail and authentication integration, etc.); and graduate and undergraduate students. The LMS Task Force participation is completely voluntary.

5. Develop a Project Charter and Project Scope: Before the LMS Task Force met for the kickoff meeting, a detailed Project Charter (https://www.dropbox.com/s/relkkcpjaquzcd9/Project%20Charter.rtf) and Project Scope (https://www.dropbox.com/s/t749u2snqfei1n4/Project%20Scope.rtf) document were developed. According to Padgett (2009), the purpose of the Project Charter is to “ensure that all the project stakeholders formally agree on the project definition and have recorded that definition in writing. The Project Charter protects against uncontrolled scope creep and the other potential missteps awaiting project teams that begin their work with only a vague or assumed understanding of the project scope and other requirements” (p. 72). Following the PMBoK project management methodology, the LMS Project Charter provided the following details:

   • business purpose for the project;

   • description;

   • objectives;

   • success criteria/expected benefits;

   • funding requirements (if any);

   • project deliverables;

   • acceptance criteria;

   • milestone schedule; and

   • approval requirements.

   A detailed LMS Project Scope document was also developed. Fairmont State University (n.d.) defines the project scope as

   describes the project's deliverables and the work required to create those deliverables. The project scope statement provides a common understanding of the project scope among all project stakeholders and describes the project's major objectives. It also enables the project team to perform more detailed planning, guides the project team's work during execution, and provides the baseline for evaluating whether requests for changes or additional work are contained within or outside the project's boundaries.

   The LMS Project Scope documentation provided the following details:

   • project scope description;

   • customer requirements;

   • statement of work;

   • project deliverables;

   • timeline;

   • acceptance criteria;

   • work breakdown structure;

   • project boundaries;

   • project assumptions;

   • initial risks and constraints; and

   • project approval.

6. Develop the Work Breakdown Structure (WBS): The WBS provides a detailed timeline of the evaluation process from the project inception through project closure. Each step in the process is mapped using a sophisticated spreadsheet that identifies timelines, critical paths and a Gantt chart that illustrates process dependencies. As each alternative LMS product is identified for review, it is mapped using the WBS. There is a critical timeline for the entire project that meets licensing requirements and school calendar year schedules. If certain milestones are not met, the entire project risks time and cost overruns, as well as the possibility of complete failure.

   The overarching WBS for this evaluation process was to narrow the field of products from six to two, from which a final LMS product selection would be made. The original six products considered were selected from evaluations conducted at other campuses. Rather than request the LMS Task Force make these initial selections (most of which they were likely unfamiliar), the Project Manager assembled the candidate products. These are all popular, viable, and proven systems used on campuses across both the United States and internationally. The timeline for review was set at six months—one product evaluation per month. Each evaluation consisted of an on-site or webinar-based presentation lasting approximately 2 hours. An evaluative rubric is used to rate each of the products following this presentation. The final two products will be evaluated in much greater detail—likely full-day demonstrations offered later in the project cycle. Additional “sandbox” sites were created for each product that allowed the faculty to export their courses from the current system and import it into the candidate system.

7. Develop a Risk Assessment Audit: Risk assessment measurement is a process used to identify and evaluate risks and their potential effect. According to the Standish Group International (2007), 65% of IT projects failed or were challenged due to poor management, cost overruns, or other environmental factors. Several possible risks were considered during the evaluative process: product licensing cost overruns; failure to include key stakeholders in the selection process; failure to provide ample information in order to make an informed decision; and failure to appropriately communicate any product decisions to the university community. In order to control for these concerns, a risk audit matrix was developed that identified areas that may compromise the project outcomes. For each risk, a mitigation strategy was developed.

8. Develop Other Measurement Criteria: Additional measurement criteria were developed in order to measure project success. Included were the following:

   • Schedule Performance Index: This is an index that measures cost of work performed against cost of work scheduled. Schedule overruns will be closely monitored throughout the project life cycle.

   • Cost Performance Index: An index that measures projected or estimated cost of work performed against actual cost of work performed. Given that this is an educational institution, the costs are normally controlled since that budgets are very restricted. If additional costs are incurred, this performance index will be utilized to measure the impact on the project.

   • Human Resource Index: An index that measures functions involving human resources management. Given that a replacement system may significantly impact human resource capital, this index is often utilized to control for variables that may impact implementation of an alternative system (RCE Associates, 2011).

   • Scope Management Control Measures: An index that ensures that the project targets the identified change and not spiral into secondary product change that may lead to cost and time overruns.

   • Quality Cost and Delivery: Three separate indexes that provide data to the project manager regarding error mitigation; waste reduction; and timely project delivery (Lean Kaizen, 2010).

9. Develop the Evaluation Rubric: Based on criteria collected from other campuses, a comprehensive evaluative rubric was developed. In order to ensure the LMS Task Force members were included in this process, a Google Doc site was established that allowed all members of the task force to add comments regarding features they wanted to see addressed in a replacement system. Google Docs proved to be the best selection for this process because users could easily add notations without the need to download the document, add comments, and reupload. That information was eventually collected and organized into feature sets. Each of these feature sets was weighted and ranked according to its importance as a product feature. As the products are evaluated, a weighted index is applied to each feature according to its relative importance to the faculty and students (e.g., the grade book is relatively more important than a chat tool in most instances). The weighting schema added the criteria ranking (1-5) with the number of scores for that feature. The result was multiplied against the criteria weighting. The sum of that result is added to the sum of the weight and divided by the sum of the weight to achieve the weighted ranking.

10. Develop a Request for Proposal and Request for Information: Once the field of products is narrowed from six to two, the LMS Task Force project manager (with assistance from the LMS Task Force) will develop a request for proposal/request for information. This process normally involves detailing the various criteria and features required in a product selection. According to OLCsoft (2004),

    An RFP should include information about your organization and project as well as questions that elicit differences among competing companies. Question types should include both ‘Yes/No’ type responses as well as open-ended text questions. An RFP should also solicit cost quotes that allow you to determine both initial and ongoing costs of the product or service.

    The various vendors then respond to that written proposal. This process reduces the potential for “vendor bias” by using a bid system, and streamlines the procurement process by identifying critical features using a rubric. The RFP normally accompanies the contract as it winds its way through the campus procurement system. Following interviews with other campuses, it was determined that the RFP should be postponed until the Task Force has narrowed its selection to the top two products. It would not be a good use of time engaging in a protracted RFP process if the product is clearly something the faculty and students reject from the onset.

11. Develop a Project Communication Plan: A comprehensive communication plan was established that provides input and feedback to and from the university community. Once the candidate products are narrowed from six to two, the university faculty, staff, and students will be invited to review the finalist products through open forums, hands-on labs, discussions, presentations, and other direct exposure mechanisms. That input will be collected using a campuswide survey designed to solicit feedback and input. Results from that survey will be communicated to the executive leadership and other administrative stakeholders. It is anticipated that the task force and the user community will reach a consensus regarding a product selection. From that consensus, pilot studies will be launched and data collected regarding the usability, features, and overall product experience. That information will also be communicated to the campus community via various channels.

    The communication mechanisms to be utilized throughout the project implementation include the following:

    • E-mail Distribution List. According to Rouse (2005)

      a distribution list is a group of mail recipients that is addressed as a single recipient. Distribution lists are used to send e-mail to groups of people without having to enter each recipient's individual address. A distribution list is different from an e-mail list in that members cannot reply to the distribution list's name to send messages to everyone else in the group.

      This communication channel provides targeted information for specific subsets of ESU employees. For example, employees directly involved with the LMS implementation will receive targeted information specific to those activities. Senior administrative sponsors will receive targeted information specific to their portion of the implementation, and so forth.

    • E-mail Project Activity Reports: Provides detailed information regarding various stages of the project. This information is more specific in nature and only intended for the LMS Task Force and Senior Administration. External stakeholders should not be privy to this level of project detail (Mehta, 2002).

    • Web Site/Project Portal: Whereas e-mail updates provide blasts of information on a regular basis, a website/project portal provides project information on an ongoing basis. This site can include images, charts, reports, and other content that is difficult to e-mail. According to DelTek (2007),

      More and more, projects and teams have become geographically dispersed. An increasing number of executives and other stakeholders want to be informed of a project's status and health in real time without wading through lengthy reports. Partners, customers and other stakeholders outside an organization also expect to be kept in the loop on a constant basis. This requires a comprehensive means of managing and communicating project information.

      Areas of the portal can be restricted to key personnel based on password access. E-mail communication logs can also be added to the site, along with discussion boards for Q&A, a project manager blog, videos, podcasts, and other features that better convey the project status. External stakeholders may be able to access certain areas of the site; however, it's really intended for the LMS Task Force and senior administration. To manage this particular project, a private Google Site was generated (Figure A). Included are all relevant documents, video recordings of the vendor presentations, and other relevant information.

    • Project Calendar: Provides project schedule information for the LMS Task Force, thus providing insight as to leads/lags/floats (project time buffers) and other variables that may affect the schedule. This information is not necessarily intended for external stakeholders and thus will not be communicated outside the project team unless a significant project delay warrants such outreach. Well-managed calendars and contact lists can be important to project productivity (Quinn, 2012).

    • Quarterly Mobile Ezine: According to eZine Marketing (2012), an electronic newsletter is a great way to promote a regular forum for a variety of information. An eZine provides project promotion information for both external and internal stakeholders. Mobile applications now represent a major communication vehicle for many types of organizations. An Ezine can be assembled relatively quickly and disseminated through electronic “marketplaces.” This information is promotional in nature—similar to a magazine. Project highlights; advantages of the new LMS; useful data; projected savings and reduced service requests can all be highlighted.

    • Project Checkpoints: These face-to-face meetings (with documentation, as deemed necessary) are generally limited to university senior administration and the Project Manager. This communication is scheduled at each major project milestone and is intended to provide an update regarding the previous completed WBS task, while providing information pertaining to the upcoming WBS task. External stakeholders will not require access to this information.

    • Decision Logs: A decision log is a list of the key decisions made in a project (Harrin, 2010). It normally includes:

      • A reference for the decision;

      • Date decision made;

      • What was agreed and why;

      • Who agreed to it; and

      • Where you can find more information or supporting documentation.

        Decision logs are used to document all project decisions and change orders. This information is limited to Project Team Leads and Senior Administration only. (No external stakeholders.) Decision logs are functional in nature and only serve as a document trail throughout the project. These logs can be stored within the web portal/intranet.

Increasingly, software applications impact broad cross-sections of users and support personnel across the enterprise. Selecting a new application or even upgrading an existing application can have a tremendous impact on the user community—both negatively and positively. Managing this process requires careful, deliberate, measured actions that take into account the needs of a variety of constituents, and plan for potential for negative consequences. Mitigating these risks, while still accomplishing the goal of changing systems in order to address the business goals of the university is one reason project management strategies are engaged. These strategies are systematic, linear, and provide controls at every level. They ensure projects of this magnitude are carried out successfully and not abandoned—a common problem for projects that do not engage a framework for change management. They also control for cost overruns and scope creep—common in large-scale software system deployment. Finally, this strategy ensures faculty, staff, and student buy-in regarding the selection.

The PMBoK proved to be the ideal mechanism for managing this process at Emporia State University. Selection of a new learning management system is a critically important process. The impact of the selection may translate to additional or sustained enrollments, improved integration of technology among the faculty, a higher quality of online courses, streamlined support, and other related benefits. A decision of this scope should never be made without participation and investment from the faculty and student community. Utilizing a project management framework ensures that milestones are met, the project remains on schedule, there is little or no scope creep, there is little or no cost overrun, and the project is well communicated to the community. All future project management at the university will likely be built around this framework.