With the large increase in the number of Web-based systems in the last decade, the computational requirements for supporting Web applications grew dramatically. Frequently an organization required dozens, hundreds, or even thousands of servers to support its customer load. (1,2) "Server farms" were created in which machines were used as Web servers, file servers, database servers, and application servers. Although not a new idea--IBM mainframes have had virtual machine capability since at least 1972 when the VM operating system (then known as VM/370) was introduced (3,4)--there has also been a recent focus on running virtual servers on powerful personal or mainframe computers. (5-9) In this paper we describe our experience at Colorado State University (CSU) running hundreds of virtual Linux ** servers on an IBM S/390 * mainframe (10) with the z/VM * (11) operating system.

The 1980s ushered in the personal computer (PC) era, which provided us with small but powerful machines that could act as servers in the client-server model of computing. The PC came at low cost and allowed end-user control of the system, without interference from a central information technology (IT) department. As client-server computing became popular, many tasks that had been traditionally performed on mainframes were migrated to the smaller and less costly PC servers. In fact, especially with the advent in the 1990s of the freely available Linux operating system and other open-source software such as the Apache Web server, server farms consisting of dozens or hundreds of these commodity PC servers were set up to service heavily used Web sites. The ability to purchase PC servers for only a few hundred dollars drove demand away from the mainframe and to these small, inexpensive, and disposable servers.

Managing large numbers of these servers, however, had its drawbacks. There had to be physical space to store them and physical wiring to interconnect them. They had to be individually installed, configured, and maintained. It could take significant time and support personnel to keep these server farms running. The low purchase cost of these PC-based servers belied the real total cost of ownership associated with operating these server farms.

In the last few years we have witnessed a return to the mainframe for specialized applications. (12-18) Using the ability of z/VM to host multiple virtual servers running a variety of guest operating systems, mainframes such as IBM's S/390 and zSeries * are now being used to create virtual server farms. (7,16,19,20,) These systems offer high processing speed and high bandwidth for I/O; the physical space they require is comparable to that for hosting small physical server farms; and it is easy to clone an initial server as many times as needed. This cloning can even be automated so that no human intervention is required to carry it out. Thus some of the problems encountered in the physical server farm environment go away. Running virtual Linux guests on a mainframe leads to space savings, easier management, and the benefit of the reliability and security of the mainframe.

One case in which virtual servers are cost-effective is when they are used as Web and database servers. (21-23) In order to support a high-volume Web site, one that is heavily accessed by large numbers of people, it is not uncommon to employ hundreds of virtual servers. In a virtual environment, new servers can be cloned on demand and a load balancer distributes incoming requests among them. Virtual servers use shared resources, such as memory and computational power, and thus are superior to physical server farms in terms of the efficient use of resources.

Because Linux can be used as the guest operating system on a z/VM virtual machine, the large collection of open-source software is readily available and can be run on such a virtual server. Server farms created in this environment can be controlled and managed efficiently on a single physical machine.

The rest of this paper is structured as follows. In the next section we introduce basic z/VM concepts. In the following three sections we describe our experience in running multiple Linux virtual servers on z/VM at CSU and focus on three types of problems we encountered: security issues, performance issues, and administration issues related to managing Linux instances. We conclude with a summary and directions for further work.

Z/VM CONCEPTS

z/VM supports the creation of virtual machines by virtualizing the resources of the computing platform: processing, communication, memory, I/O, and networking resources. Although virtual machines share physical resources, they are isolated from each other, and each virtual machine provides the illusion of access to the entire computer. Each virtual machine hosts a guest operating system such as z/OS *, OS/390 *, CMS, and Linux. The guests operate completely independently of one another.

Virtualization, such as that provided by z/VM, is often used together with partitioning. On computing platforms such as the S/390 and the z/Series, the hardware supports logical partitioning (a logical partition is referred to as an LPAR) of the machine into several distinct and isolated regions that operate independently of one another. Each unit of physical resource is thus mapped into a specific partition.

Partitioning and virtualization can be viewed as complementary technologies. Figure 1 illustrates a range of possible configurations for running Linux on an S/390 or zSeries mainframe that involve virtual machines and LPARs.

[FIGURE 1 OMITTED]

In Figure 1A Linux is running natively on the mainframe, that is, without logical partitioning. This option is rarely used because it provides for a single operating-system instance on the entire mainframe. In fact, it is no longer possible to configure zSeries mainframes without LPARs--the IBM eServer * zSeries 900 (z900) series was the last family of processors that supported this option. In Figure 1B the machine is configured with three LPARs, and Linux is running natively within each LPAR, as before. In this configuration no virtual machines are created; Linux is running natively and not as a guest operating system on a virtual machine. The maximum number of LPARs depends on the hardware and varies between 15 and 30. Figure 1C depicts a configuration without partitions in which z/VM supports any number of Linux guests (three guests are shown in the figure). The fourth scenario illustrated in Figure 1D combines logical partitioning and virtual machines. The figure shows five LPARs; three of these host operating systems are running natively; whereas, the other two partitions host z/ VM instances that support a number of virtual machines. Note that the operating systems running either in an LPAR or on a z/VM-supported virtual machine need not be the same.

The fourth option is similar to the design used for the CSU system. It operates within one of the LPARs and has approximately 500 virtual Linux-server z/VM guests running within this LPAR. Anywhere between 50 and 300 of these are typically active at any given time. At the time the work described here was performed, we were using Version 4.4 of z/VM, running on S/390 hardware, with almost all of the Linux instances running SLES 8 (SuSE Linux Enterprise Server Version 8) and only a few running SLES 7. Although our platform was S/390, most of the results should apply to the zSeries platform as well.

SECURITY ISSUES

In this section we focus on three security-related issues: account passwords, ssh (secure shell) keys, and software updates. Although these are important issues for all servers, for systems with large numbers of cloned virtual servers they become critical.

Account passwords

Access to a Linux instance typically requires an account and a password. In an environment with hundreds of Linux instances, account maintenance can become extremely time-consuming, and thus some form of automated maintenance is needed.

Although a z/VM administrator has the ability to issue commands and push out updates to all Linux instances (z/VM guests), using this approach has its drawbacks. Possibly the biggest problem is the time delay involved. The Linux administrator must determine what updates are needed, request the z/VM administrator to perform the updates, wait for this task to get done, and then inform the Linux-instance users that the updated machines are ready for use. Relying on the z/VM administrator to carry out these updates clearly lengthens the process.