The Experts below are selected from a list of 267 Experts worldwide ranked by ideXlab platform

Philipp Leitner - One of the best experts on this subject based on the ideXlab platform.

  • Software microbenchmarking in the Cloud. How bad is it really?
    Empirical Software Engineering, 2019
    Co-Authors: Christoph Laaber, Joel Scheuner, Philipp Leitner
    Abstract:

    Rigorous performance engineering traditionally assumes measuring on bare-metal environments to control for as many confounding factors as possible. Unfortunately, some researchers and practitioners might not have access, knowledge, or funds to operate dedicated performance-testing hardware, making public Clouds an attractive alternative. However, shared public Cloud environments are inherently unpredictable in terms of the system performance they provide. In this study, we explore the effects of Cloud environments on the variability of performance test results and to what extent slowdowns can still be reliably detected even in a public Cloud. We focus on software microbenchmarks as an example of performance tests and execute extensive experiments on three different well-known public Cloud services (AWS, GCE, and Azure) using three different Cloud Instance types per service. We also compare the results to a hosted bare-metal offering from IBM Bluemix. In total, we gathered more than 4.5 million unique microbenchmarking data points from benchmarks written in Java and Go. We find that the variability of results differs substantially between benchmarks and Instance types (by a coefficient of variation from 0.03% to > 100%). However, executing test and control experiments on the same Instances (in randomized order) allows us to detect slowdowns of 10% or less with high confidence, using state-of-the-art statistical tests (i.e., Wilcoxon rank-sum and overlapping bootstrapped confidence intervals). Finally, our results indicate that Wilcoxon rank-sum manages to detect smaller slowdowns in Cloud environments.

  • An Approach and Case Study of Cloud Instance Type Selection for Multi-Tier Web Applications
    Proceedings - 2017 17th IEEE ACM International Symposium on Cluster Cloud and Grid Computing CCGRID 2017, 2017
    Co-Authors: Christian Davatz, Joel Scheuner, Christian Inzinger, Philipp Leitner
    Abstract:

    A challenging problem for users of Infrastructure-as-a-Service (IaaS) Clouds is selecting Cloud providers, regions, and Instance types cost-optimally for a given desired service level. Issues such as hardware heterogeneity, contention, and virtual machine (VM) placement can result in considerably differing performance across supposedly equivalent Cloud resources. Existing research on Cloud benchmarking helps, but often the focus is on providing low-level microbenchmarks (e.g., CPU or network speed), which are hard to map to concrete business metrics of enterprise Cloud applications, such as request throughput of a multi-tier Web application. In this paper, we propose Okta, a general approach for fairly and comprehensively benchmarking the performance and cost of a multi-tier Web application hosted in an IaaS Cloud. We exemplify our approach for a case study based on the two-tier AcmeAir application, which we evaluate for 11 real-life deployment configurations on Amazon EC2 and Google Compute Engine. Our results show that for this application, choosing compute-optimized Instance types in the Web layer and small bursting Instances for the database tier leads to the overall most cost-effective deployments. This result held true for both Cloud providers. The least cost-effective configuration in our study provides only about 67% of throughput per US dollar spent. Our case study can serve as a blueprint for future industrial or academic application benchmarking projects.

  • CCGrid - An Approach and Case Study of Cloud Instance Type Selection for Multi-Tier Web Applications
    2017 17th IEEE ACM International Symposium on Cluster Cloud and Grid Computing (CCGRID), 2017
    Co-Authors: Christian Davatz, Joel Scheuner, Christian Inzinger, Philipp Leitner
    Abstract:

    A challenging problem for users of Infrastructure-as-a-Service (IaaS) Clouds is selecting Cloud providers, regions, and Instance types cost-optimally for a given desired service level. Issues such as hardware heterogeneity, contention, and virtual machine (VM) placement can result in considerably differing performance across supposedly equivalent Cloud resources. Existing research on Cloud benchmarking helps, but often the focus is on providing low-level microbenchmarks (e.g., CPU or network speed), which are hard to map to concrete business metrics of enterprise Cloud applications, such as request throughput of a multi-tier Web application. In this paper, we propose Okta, a general approach for fairly and comprehensively benchmarking the performance and cost of a multi-tier Web application hosted in an IaaS Cloud. We exemplify our approach for a case study based on the two-tier AcmeAir application, which we evaluate for 11 real-life deployment configurations on Amazon EC2 and Google Compute Engine. Our results show that for this application, choosing compute-optimized Instance types in the Web layer and small bursting Instances for the database tier leads to the overall most cost-effective deployments. This result held true for both Cloud providers. The least cost-effective configuration in our study provides only about 67% of throughput per US dollar spent. Our case study can serve as a blueprint for future industrial or academic application benchmarking projects.

  • UCC - Bursting with possibilities: an empirical study of credit-based bursting Cloud Instance types
    2015
    Co-Authors: Philipp Leitner, Joel Scheuner
    Abstract:

    We study the performance and cost efficiency as perceived by the end user of a specific class of Infrastructure-as-a-Service (IaaS) Cloud Instances, namely credit-based bursting Instances. This class of Instance types has been introduced by Amazon EC2 in summer 2014, and behaves on a fundamental level differently than any other existing Instance type, either from EC2 or other vendors. We introduce a basic formal model for fostering the understanding and analysis of these types, and empirically study their performance in practice. Further, we compare the performance of credit-based bursting Cloud Instance types to existing general-purpose types, and derive potential use cases for practitioners. Our results indicate that bursting Instance types are cost-efficient for CPU-bound applications with an average utilization of less than 40%, as well as for non-critical IO-bound applications. Finally, we also discuss a simple boosting scheme that enables practitioners to improve the cost efficiency of their bursting Instance usage under given constraints.

  • Bursting with Possibilities-An Empirical Study of Credit-Based Bursting Cloud Instance Types
    Proceedings - 2015 IEEE ACM 8th International Conference on Utility and Cloud Computing UCC 2015, 2015
    Co-Authors: Philipp Leitner, Joel Scheuner
    Abstract:

    We study the performance and cost efficiency as perceived by the end user of a specific class of Infrastructure-as-a-Service (IaaS) Cloud Instances, namely credit-based bursting Instances. This class of Instance types has been introduced by Amazon EC2 in summer 2014, and behaves on a fundamental level differently than any other existing Instance type, either from EC2 or other vendors. We introduce a basic formal model for fostering the understanding and analysis of these types, and empirically study their performance in practice. Further, we compare the performance of credit-based bursting Cloud Instance types to existing general-purpose types, and derive potential use cases for practitioners. Our results indicate that bursting Instance types are cost-efficient for CPU-bound applications with an average utilization of less than 40%, as well as for non-critical IO-bound applications. Finally, we also discuss a simple boosting scheme that enables practitioners to improve the cost efficiency of their bursting Instance usage under given constraints.

Christian Davatz - One of the best experts on this subject based on the ideXlab platform.

  • An Approach and Case Study of Cloud Instance Type Selection for Multi-Tier Web Applications
    Proceedings - 2017 17th IEEE ACM International Symposium on Cluster Cloud and Grid Computing CCGRID 2017, 2017
    Co-Authors: Christian Davatz, Joel Scheuner, Christian Inzinger, Philipp Leitner
    Abstract:

    A challenging problem for users of Infrastructure-as-a-Service (IaaS) Clouds is selecting Cloud providers, regions, and Instance types cost-optimally for a given desired service level. Issues such as hardware heterogeneity, contention, and virtual machine (VM) placement can result in considerably differing performance across supposedly equivalent Cloud resources. Existing research on Cloud benchmarking helps, but often the focus is on providing low-level microbenchmarks (e.g., CPU or network speed), which are hard to map to concrete business metrics of enterprise Cloud applications, such as request throughput of a multi-tier Web application. In this paper, we propose Okta, a general approach for fairly and comprehensively benchmarking the performance and cost of a multi-tier Web application hosted in an IaaS Cloud. We exemplify our approach for a case study based on the two-tier AcmeAir application, which we evaluate for 11 real-life deployment configurations on Amazon EC2 and Google Compute Engine. Our results show that for this application, choosing compute-optimized Instance types in the Web layer and small bursting Instances for the database tier leads to the overall most cost-effective deployments. This result held true for both Cloud providers. The least cost-effective configuration in our study provides only about 67% of throughput per US dollar spent. Our case study can serve as a blueprint for future industrial or academic application benchmarking projects.

  • CCGrid - An Approach and Case Study of Cloud Instance Type Selection for Multi-Tier Web Applications
    2017 17th IEEE ACM International Symposium on Cluster Cloud and Grid Computing (CCGRID), 2017
    Co-Authors: Christian Davatz, Joel Scheuner, Christian Inzinger, Philipp Leitner
    Abstract:

    A challenging problem for users of Infrastructure-as-a-Service (IaaS) Clouds is selecting Cloud providers, regions, and Instance types cost-optimally for a given desired service level. Issues such as hardware heterogeneity, contention, and virtual machine (VM) placement can result in considerably differing performance across supposedly equivalent Cloud resources. Existing research on Cloud benchmarking helps, but often the focus is on providing low-level microbenchmarks (e.g., CPU or network speed), which are hard to map to concrete business metrics of enterprise Cloud applications, such as request throughput of a multi-tier Web application. In this paper, we propose Okta, a general approach for fairly and comprehensively benchmarking the performance and cost of a multi-tier Web application hosted in an IaaS Cloud. We exemplify our approach for a case study based on the two-tier AcmeAir application, which we evaluate for 11 real-life deployment configurations on Amazon EC2 and Google Compute Engine. Our results show that for this application, choosing compute-optimized Instance types in the Web layer and small bursting Instances for the database tier leads to the overall most cost-effective deployments. This result held true for both Cloud providers. The least cost-effective configuration in our study provides only about 67% of throughput per US dollar spent. Our case study can serve as a blueprint for future industrial or academic application benchmarking projects.

Christian Inzinger - One of the best experts on this subject based on the ideXlab platform.

  • An Approach and Case Study of Cloud Instance Type Selection for Multi-Tier Web Applications
    Proceedings - 2017 17th IEEE ACM International Symposium on Cluster Cloud and Grid Computing CCGRID 2017, 2017
    Co-Authors: Christian Davatz, Joel Scheuner, Christian Inzinger, Philipp Leitner
    Abstract:

    A challenging problem for users of Infrastructure-as-a-Service (IaaS) Clouds is selecting Cloud providers, regions, and Instance types cost-optimally for a given desired service level. Issues such as hardware heterogeneity, contention, and virtual machine (VM) placement can result in considerably differing performance across supposedly equivalent Cloud resources. Existing research on Cloud benchmarking helps, but often the focus is on providing low-level microbenchmarks (e.g., CPU or network speed), which are hard to map to concrete business metrics of enterprise Cloud applications, such as request throughput of a multi-tier Web application. In this paper, we propose Okta, a general approach for fairly and comprehensively benchmarking the performance and cost of a multi-tier Web application hosted in an IaaS Cloud. We exemplify our approach for a case study based on the two-tier AcmeAir application, which we evaluate for 11 real-life deployment configurations on Amazon EC2 and Google Compute Engine. Our results show that for this application, choosing compute-optimized Instance types in the Web layer and small bursting Instances for the database tier leads to the overall most cost-effective deployments. This result held true for both Cloud providers. The least cost-effective configuration in our study provides only about 67% of throughput per US dollar spent. Our case study can serve as a blueprint for future industrial or academic application benchmarking projects.

  • CCGrid - An Approach and Case Study of Cloud Instance Type Selection for Multi-Tier Web Applications
    2017 17th IEEE ACM International Symposium on Cluster Cloud and Grid Computing (CCGRID), 2017
    Co-Authors: Christian Davatz, Joel Scheuner, Christian Inzinger, Philipp Leitner
    Abstract:

    A challenging problem for users of Infrastructure-as-a-Service (IaaS) Clouds is selecting Cloud providers, regions, and Instance types cost-optimally for a given desired service level. Issues such as hardware heterogeneity, contention, and virtual machine (VM) placement can result in considerably differing performance across supposedly equivalent Cloud resources. Existing research on Cloud benchmarking helps, but often the focus is on providing low-level microbenchmarks (e.g., CPU or network speed), which are hard to map to concrete business metrics of enterprise Cloud applications, such as request throughput of a multi-tier Web application. In this paper, we propose Okta, a general approach for fairly and comprehensively benchmarking the performance and cost of a multi-tier Web application hosted in an IaaS Cloud. We exemplify our approach for a case study based on the two-tier AcmeAir application, which we evaluate for 11 real-life deployment configurations on Amazon EC2 and Google Compute Engine. Our results show that for this application, choosing compute-optimized Instance types in the Web layer and small bursting Instances for the database tier leads to the overall most cost-effective deployments. This result held true for both Cloud providers. The least cost-effective configuration in our study provides only about 67% of throughput per US dollar spent. Our case study can serve as a blueprint for future industrial or academic application benchmarking projects.

Joel Scheuner - One of the best experts on this subject based on the ideXlab platform.

  • Software microbenchmarking in the Cloud. How bad is it really?
    Empirical Software Engineering, 2019
    Co-Authors: Christoph Laaber, Joel Scheuner, Philipp Leitner
    Abstract:

    Rigorous performance engineering traditionally assumes measuring on bare-metal environments to control for as many confounding factors as possible. Unfortunately, some researchers and practitioners might not have access, knowledge, or funds to operate dedicated performance-testing hardware, making public Clouds an attractive alternative. However, shared public Cloud environments are inherently unpredictable in terms of the system performance they provide. In this study, we explore the effects of Cloud environments on the variability of performance test results and to what extent slowdowns can still be reliably detected even in a public Cloud. We focus on software microbenchmarks as an example of performance tests and execute extensive experiments on three different well-known public Cloud services (AWS, GCE, and Azure) using three different Cloud Instance types per service. We also compare the results to a hosted bare-metal offering from IBM Bluemix. In total, we gathered more than 4.5 million unique microbenchmarking data points from benchmarks written in Java and Go. We find that the variability of results differs substantially between benchmarks and Instance types (by a coefficient of variation from 0.03% to > 100%). However, executing test and control experiments on the same Instances (in randomized order) allows us to detect slowdowns of 10% or less with high confidence, using state-of-the-art statistical tests (i.e., Wilcoxon rank-sum and overlapping bootstrapped confidence intervals). Finally, our results indicate that Wilcoxon rank-sum manages to detect smaller slowdowns in Cloud environments.

  • An Approach and Case Study of Cloud Instance Type Selection for Multi-Tier Web Applications
    Proceedings - 2017 17th IEEE ACM International Symposium on Cluster Cloud and Grid Computing CCGRID 2017, 2017
    Co-Authors: Christian Davatz, Joel Scheuner, Christian Inzinger, Philipp Leitner
    Abstract:

    A challenging problem for users of Infrastructure-as-a-Service (IaaS) Clouds is selecting Cloud providers, regions, and Instance types cost-optimally for a given desired service level. Issues such as hardware heterogeneity, contention, and virtual machine (VM) placement can result in considerably differing performance across supposedly equivalent Cloud resources. Existing research on Cloud benchmarking helps, but often the focus is on providing low-level microbenchmarks (e.g., CPU or network speed), which are hard to map to concrete business metrics of enterprise Cloud applications, such as request throughput of a multi-tier Web application. In this paper, we propose Okta, a general approach for fairly and comprehensively benchmarking the performance and cost of a multi-tier Web application hosted in an IaaS Cloud. We exemplify our approach for a case study based on the two-tier AcmeAir application, which we evaluate for 11 real-life deployment configurations on Amazon EC2 and Google Compute Engine. Our results show that for this application, choosing compute-optimized Instance types in the Web layer and small bursting Instances for the database tier leads to the overall most cost-effective deployments. This result held true for both Cloud providers. The least cost-effective configuration in our study provides only about 67% of throughput per US dollar spent. Our case study can serve as a blueprint for future industrial or academic application benchmarking projects.

  • CCGrid - An Approach and Case Study of Cloud Instance Type Selection for Multi-Tier Web Applications
    2017 17th IEEE ACM International Symposium on Cluster Cloud and Grid Computing (CCGRID), 2017
    Co-Authors: Christian Davatz, Joel Scheuner, Christian Inzinger, Philipp Leitner
    Abstract:

    A challenging problem for users of Infrastructure-as-a-Service (IaaS) Clouds is selecting Cloud providers, regions, and Instance types cost-optimally for a given desired service level. Issues such as hardware heterogeneity, contention, and virtual machine (VM) placement can result in considerably differing performance across supposedly equivalent Cloud resources. Existing research on Cloud benchmarking helps, but often the focus is on providing low-level microbenchmarks (e.g., CPU or network speed), which are hard to map to concrete business metrics of enterprise Cloud applications, such as request throughput of a multi-tier Web application. In this paper, we propose Okta, a general approach for fairly and comprehensively benchmarking the performance and cost of a multi-tier Web application hosted in an IaaS Cloud. We exemplify our approach for a case study based on the two-tier AcmeAir application, which we evaluate for 11 real-life deployment configurations on Amazon EC2 and Google Compute Engine. Our results show that for this application, choosing compute-optimized Instance types in the Web layer and small bursting Instances for the database tier leads to the overall most cost-effective deployments. This result held true for both Cloud providers. The least cost-effective configuration in our study provides only about 67% of throughput per US dollar spent. Our case study can serve as a blueprint for future industrial or academic application benchmarking projects.

  • UCC - Bursting with possibilities: an empirical study of credit-based bursting Cloud Instance types
    2015
    Co-Authors: Philipp Leitner, Joel Scheuner
    Abstract:

    We study the performance and cost efficiency as perceived by the end user of a specific class of Infrastructure-as-a-Service (IaaS) Cloud Instances, namely credit-based bursting Instances. This class of Instance types has been introduced by Amazon EC2 in summer 2014, and behaves on a fundamental level differently than any other existing Instance type, either from EC2 or other vendors. We introduce a basic formal model for fostering the understanding and analysis of these types, and empirically study their performance in practice. Further, we compare the performance of credit-based bursting Cloud Instance types to existing general-purpose types, and derive potential use cases for practitioners. Our results indicate that bursting Instance types are cost-efficient for CPU-bound applications with an average utilization of less than 40%, as well as for non-critical IO-bound applications. Finally, we also discuss a simple boosting scheme that enables practitioners to improve the cost efficiency of their bursting Instance usage under given constraints.

  • Bursting with Possibilities-An Empirical Study of Credit-Based Bursting Cloud Instance Types
    Proceedings - 2015 IEEE ACM 8th International Conference on Utility and Cloud Computing UCC 2015, 2015
    Co-Authors: Philipp Leitner, Joel Scheuner
    Abstract:

    We study the performance and cost efficiency as perceived by the end user of a specific class of Infrastructure-as-a-Service (IaaS) Cloud Instances, namely credit-based bursting Instances. This class of Instance types has been introduced by Amazon EC2 in summer 2014, and behaves on a fundamental level differently than any other existing Instance type, either from EC2 or other vendors. We introduce a basic formal model for fostering the understanding and analysis of these types, and empirically study their performance in practice. Further, we compare the performance of credit-based bursting Cloud Instance types to existing general-purpose types, and derive potential use cases for practitioners. Our results indicate that bursting Instance types are cost-efficient for CPU-bound applications with an average utilization of less than 40%, as well as for non-critical IO-bound applications. Finally, we also discuss a simple boosting scheme that enables practitioners to improve the cost efficiency of their bursting Instance usage under given constraints.

Silvery Fu - One of the best experts on this subject based on the ideXlab platform.

  • Truthful Online Auction for Cloud Instance Subletting
    2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS), 2017
    Co-Authors: Silvery Fu
    Abstract:

    Despite that IaaS users are busy scaling up/out their Cloud Instances to meet the ever-increasing demands, the dynamics of their demands, as well as the coarse-grained billing options offered by leading Cloud providers, have led to substantial Instance underutilization in both temporal and spatial domains. This paper theoretically examines an Instance subletting service, where underutilized Instances are leased to others within user-specified periods. Serving as a secondary market that complements the existing Instance market of IaaS providers,we specifically identify the theoretical challenges in Instance subletting services, and design an online auction mechanism to make allocation and pricing decisions for the Instances to be sublet. Our mechanism guarantees truthfulness and individual rationality with the best possible competitive ratio. Extensive trace-driven simulations show that our proposed mechanism achieves significant performance gains in both cost and social welfare.