Dissertation Abstract

Black Box Methods for Inferring Parallel Applications’ Properties in Virtual Environments

Author: Ashish Gupta

Adviser: Prof. Peter Dinda

Virtual machine based distributed computing greatly simplifies and enhances adaptive/autonomic computing by lowering the level of abstraction, benefiting both resource providers and users. We are developing Virtuoso, a middleware system for virtual machine shared-resource computing (e.g. grids) that provides numerous advantages and overcomes many obstacles a user faces in using shared resources for deploying distributed applications. A major hurdle for distributed applications to function in such an environment is locating, reserving/scheduling and dynamically adapting to the appropriate communication and computational resources so as to meet the applications’ demands, limited by cost constraints. Resources can be very heterogeneous, especially in wide area or shared infrastructures, and their availability is also highly dynamic.

To achieve such automated adaptation, one must first learn about the various demands and properties of the distributed application running inside the VMs. My thesis is that it is feasible to infer the applications’ demands and behavior, to a significant degree, without actually knowing much about the application or its operating system; I have investigated and demonstrated numerous novel techniques to infer many useful properties of parallel applications, in an automated fashion. Throughout my work I have used a black box approach to inference. Thus the results are applicable to existing, unmodified applications and operating systems and are widely applicable.

I show how to infer the communication behavior and the runtime topology of a parallel application. I also show how to infer very useful runtime properties of a parallel application such as its runtime performance. I have developed the algorithms to understand quantitatively the slowdown of an application under external load to find global bottlenecks at the resource and process level using time decomposition of the application’s execution.

In addition, my dissertation also gives limited evidence and points to other work that shows that this inferred information can actually be used to benefit the application, without having any knowledge of the application/OS itself.

Publications related to the dissertation

A. Gupta, M. Zangrilli, A. Sundararaj, A. Huang, P. Dinda, and B. Lowekamp, Free Network Measurment for Adaptive Virtualized Distributed Computing, Proceedings of the 20th International Parallel and Distributed Processing Symposium (IPDPS 2006), April, 2006. pdf A poster appeared at ACM/IEEE SC 2005, and a previous version is available as Technical Report NWU-CS-05-13.

A. Sundararaj, A. Gupta, and P. Dinda, Increasing Application Performance In Virtual Environments Through Run-time Inference and Adaptation, Proceedings of the 14th IEEE International Symposium on High Performance Distributed Computing (HPDC 2005). pdf

A. Sundararaj, A. Gupta, P. Dinda, Dynamic Topology Adaptation of Virtual Networks of Virtual Machines, Proceedings of the Seventh Workshop on Langauges, Compilers and Run-time Support for Scalable Systems (LCR 2004). pdf

A. Gupta, P. Dinda, Inferring the Topology and Traffic Load of Parallel Programs Running In a Virtual Machine Environment, Proceedings of the 10th Workshop on Job Scheduling Policies for Parallel Processing, June, 2004, pdf