The Experts below are selected from a list of 12909 Experts worldwide ranked by ideXlab platform
Juan Carlos Fernández - One of the best experts on this subject based on the ideXlab platform.
-
Performance Model of MapReduce Iterative Applications for Hybrid Cloud Bursting
IEEE Transactions on Parallel and Distributed Systems, 2018Co-Authors: Francisco J. Clemente-castelló, Rafael Mayo, Bogdan Nicolae, Juan Carlos FernándezAbstract:Hybrid Cloud bursting (i.e., leasing temporary off-premise Cloud resources to boost the overall capacity during peak utilization) can be a cost-effective way to deal with the increasing complexity of big data analytics, especially for iterative applications. However, the low through-put, high latency network link between the on-premise and off-premise resources ("weak link") makes maintaining scalability difficult. While several data locality techniques have been designed for big data bursting on Hybrid Clouds, their effectiveness is difficult to estimate in advance. Yet such estimations are critical, because they help users decide whether the extra pay-as-you-go cost incurred by using the off-premise resources justifies the runtime speed-up. To this end, the current paper presents a performance model and methodology to estimate the runtime of iterative MapReduce applications in a Hybrid Cloud-bursting scenario. The paper focuses on the overhead incurred by the weak link at fine granularity, for both the map and the reduce phases. This approach enables high estimation accuracy, as demonstrated by extensive experiments at scale using a mix of real-world iterative MapReduce applications from standard big data benchmarking suites that cover a broad spectrum of data patterns. Not only are the produced estimations accurate in absolute terms compared with experimental results, but they are also up to an order of magnitude more accurate than applying state-of-art estimation approaches originally designed for single-site MapReduce deployments.
-
Cost Model and Analysis of Iterative MapReduce Applications for Hybrid Cloud Bursting
2017 17th IEEE ACM International Symposium on Cluster Cloud and Grid Computing (CCGRID), 2017Co-Authors: Francisco J. Clemente-castelló, Rafael Mayo, Juan Carlos FernándezAbstract:A popular and cost-effective way to deal with the increasing complexity of big data analytics is Hybrid Cloud bursting that leases temporary off-premise Cloud resources to boost the overall capacity during peak utilization. The main challenge of Hybrid Cloud bursting is that the network link between the on-premise and the off-premise computational resources often exhibit high latency and low throughput ("weak link") compared to the links within the same data-center. This paper introduces a cost model that is specifically designed for iterative MapReduce applications running in a Hybrid Cloud bursting scenario, which are a popular class of large-scale data-intensive applications that provides near real-time responsiveness. Using this cost model, users can discover trends that can be leveraged to reason about how to balance performance, accuracy and cost such that it op-timizes their requirements. We illustrated this approach through a cost analysis that focuses on two real-life iterative MapReduce applications using extensive horizontal scalability experiments that involve multiple Hybrid Cloud bursting strategies.
-
Evaluation of Data Locality Strategies for Hybrid Cloud Bursting of Iterative MapReduce
2017Co-Authors: Francisco J. Clemente-castelló, Rafael Mayo, M. Mustafa Rafique, Bogdan Nicolae, Juan Carlos FernándezAbstract:Hybrid Cloud bursting (i.e., leasing temporary off-premise Cloud resources to boost the overall capacity during peak utilization) is a popular and cost-effective way to deal with the increasing complexity of big data analytics. It is particularly promising for iterative MapReduce applications that reuse massive amounts of input data at each iteration, which compensates for the high overhead and cost of concurrent data transfers from the on-premise to the off-premise VMs over a weak inter-site link that is of limited capacity. In this paper we study how to combine various MapReduce data locality techniques designed for Hybrid Cloud bursting in order to achieve scalability for iterative MapReduce applications in a cost-effective fashion. This is a non-trivial problem due to the complex interaction between the data movements over the weak link and the scheduling of computational tasks that have to adapt to the shifting data distribution. We show that using the right combination of techniques, iterative MapReduce applications can scale well in a Hybrid Cloud bursting scenario and come even close to the scalability observed in single sites.
Rajkumar Buyya - One of the best experts on this subject based on the ideXlab platform.
-
shared data aware dynamic resource provisioning and task scheduling for data intensive applications on Hybrid Clouds using aneka
Future Generation Computer Systems, 2020Co-Authors: Rajinder Sandhu, Shreshth Tuli, Rajkumar BuyyaAbstract:Abstract In the recent years, data-intensive applications have been growing at an increasing rate and there is a critical need to solve the high-performance and scalability issues. Hybrid Cloud Computing paradigm provides a promising solution to harness local infrastructure and remote resources and provide high Quality of Service (QoS) for time sensitive and data-intensive applications. Generally, Hybrid Cloud deployments have a heterogeneous pool of resources and it becomes a challenging task to efficiently utilize resources to provide optimum results. In modern data hungry applications, it is crucial to optimize bandwidth consumption, latency and networking overheads. Moreover, most of them have large extent of file sharing capability. The existing algorithms do not explicitly consider file sharing scenarios that leads large data transmission times and has severe effects on latency. In this direction, this paper focuses on building upon existing dynamic resource provisioning and task scheduling algorithms to provide better QoS in Hybrid Cloud environments for data intensive applications in a shared file task environment. The efficiency of proposed algorithms is demonstrated by deploying them on Microsoft Azure using Aneka, a platform for developing scalable applications on the Cloud. Experiments using real-world applications and datasets show that proposed algorithms are able to allocate tasks and extend to public Cloud resources more efficiently, reducing deadline violations and improving response times to give response time reduction of upto 40.12% for a sample local alignment search application on genome sequences.
-
virtual networking with azure for Hybrid Cloud computing in aneka
IEEE International Conference on Cloud Computing Technology and Science, 2017Co-Authors: Adel Nadjaran Toosi, Rajkumar BuyyaAbstract:Hybrid Cloud environments are a highly scalable and cost-effective option for enterprises that need to expand their on-premises infrastructure. In every Hybrid Cloud solutions, the issue of inter-Cloud network connectivity has to be overcome to allow communications, possibly secure, between resources scattered over multiple networks. Network visualization provides the right method for addressing this issue. We present how Azure Virtual Private Network (VPN) services are used to establish an overlay network for Hybrid Clouds in our Aneka platform. First, we explain how Aneka resource provisioning module is extended to support Azure Resource Manger (ARM) application programming interfaces (APIs). Then, we walk through the process of establishment of an Azure Point-to-Site VPN to provide connectivity between Aneka nodes in the Hybrid Cloud environment. Finally, we present a case study Hybrid Cloud in Aneka and we experiment with it to demonstrate the functionality of the system.
-
failure aware resource provisioning for Hybrid Cloud infrastructure
Journal of Parallel and Distributed Computing, 2012Co-Authors: Bahman Javadi, Jemal Abawajy, Rajkumar BuyyaAbstract:Hybrid Cloud computing is receiving increasing attention in recent days. In order to realize the full potential of the Hybrid Cloud platform, an architectural framework for efficiently coupling public and private Clouds is necessary. As resource failures due to the increasing functionality and complexity of Hybrid Cloud computing are inevitable, a failure-aware resource provisioning algorithm that is capable of attending to the end-users quality of service (QoS) requirements is paramount. In this paper, we propose a scalable Hybrid Cloud infrastructure as well as resource provisioning policies to assure QoS targets of the users. The proposed policies take into account the workload model and the failure correlations to redirect users' requests to the appropriate Cloud providers. Using real failure traces and a workload model, we evaluate the proposed resource provisioning policies to demonstrate their performance, cost as well as performance-cost efficiency. Simulation results reveal that in a realistic working condition while adopting user estimates for the requests in the provisioning policies, we are able to improve the users' QoS about 32% in terms of deadline violation rate and 57% in terms of slowdown with a limited cost on a public Cloud.
Hong Zhong - One of the best experts on this subject based on the ideXlab platform.
-
firework data processing and sharing for Hybrid Cloud edge analytics
IEEE Transactions on Parallel and Distributed Systems, 2018Co-Authors: Quan Zhang, Weisong Shi, Qingyang Zhang, Hong ZhongAbstract:Now we are entering the era of the Internet of Everything (IoE) and billions of sensors and actuators are connected to the network. As one of the most sophisticated IoE applications, real-time video analytics is promising to significantly improve public safety, business intelligence, and healthcare & life science, among others. However, Cloud-centric video analytics requires that all video data must be preloaded to a centralized cluster or the Cloud, which suffers from high response latency and high cost of data transmission, given the scale of zettabytes of video data generated by IoE devices. Moreover, video data is rarely shared among multiple stakeholders due to various concerns, which restricts the practical deployment of video analytics that takes advantages of many data sources to make smart decisions. Furthermore, there is no efficient programming interface for developers and users to easily program and deploy IoE applications across geographically distributed computation resources. In this paper, we present a new computing framework, Firework , which facilitates distributed data processing and sharing for IoE applications via a virtual shared data view and service composition. We designed an easy-to-use programming interface for Firework to allow developers to program on Firework . This paper describes the system design, implementation, and programming interface of Firework . The experimental results of a video analytics application demonstrate that Firework reduces up to 19.52 percent of response latency and at least 72.77 percent of network bandwidth cost, compared to a Cloud-centric solution.
Francisco J. Clemente-castelló - One of the best experts on this subject based on the ideXlab platform.
-
Performance Model of MapReduce Iterative Applications for Hybrid Cloud Bursting
IEEE Transactions on Parallel and Distributed Systems, 2018Co-Authors: Francisco J. Clemente-castelló, Rafael Mayo, Bogdan Nicolae, Juan Carlos FernándezAbstract:Hybrid Cloud bursting (i.e., leasing temporary off-premise Cloud resources to boost the overall capacity during peak utilization) can be a cost-effective way to deal with the increasing complexity of big data analytics, especially for iterative applications. However, the low through-put, high latency network link between the on-premise and off-premise resources ("weak link") makes maintaining scalability difficult. While several data locality techniques have been designed for big data bursting on Hybrid Clouds, their effectiveness is difficult to estimate in advance. Yet such estimations are critical, because they help users decide whether the extra pay-as-you-go cost incurred by using the off-premise resources justifies the runtime speed-up. To this end, the current paper presents a performance model and methodology to estimate the runtime of iterative MapReduce applications in a Hybrid Cloud-bursting scenario. The paper focuses on the overhead incurred by the weak link at fine granularity, for both the map and the reduce phases. This approach enables high estimation accuracy, as demonstrated by extensive experiments at scale using a mix of real-world iterative MapReduce applications from standard big data benchmarking suites that cover a broad spectrum of data patterns. Not only are the produced estimations accurate in absolute terms compared with experimental results, but they are also up to an order of magnitude more accurate than applying state-of-art estimation approaches originally designed for single-site MapReduce deployments.
-
Cost Model and Analysis of Iterative MapReduce Applications for Hybrid Cloud Bursting
2017 17th IEEE ACM International Symposium on Cluster Cloud and Grid Computing (CCGRID), 2017Co-Authors: Francisco J. Clemente-castelló, Rafael Mayo, Juan Carlos FernándezAbstract:A popular and cost-effective way to deal with the increasing complexity of big data analytics is Hybrid Cloud bursting that leases temporary off-premise Cloud resources to boost the overall capacity during peak utilization. The main challenge of Hybrid Cloud bursting is that the network link between the on-premise and the off-premise computational resources often exhibit high latency and low throughput ("weak link") compared to the links within the same data-center. This paper introduces a cost model that is specifically designed for iterative MapReduce applications running in a Hybrid Cloud bursting scenario, which are a popular class of large-scale data-intensive applications that provides near real-time responsiveness. Using this cost model, users can discover trends that can be leveraged to reason about how to balance performance, accuracy and cost such that it op-timizes their requirements. We illustrated this approach through a cost analysis that focuses on two real-life iterative MapReduce applications using extensive horizontal scalability experiments that involve multiple Hybrid Cloud bursting strategies.
-
Evaluation of Data Locality Strategies for Hybrid Cloud Bursting of Iterative MapReduce
2017Co-Authors: Francisco J. Clemente-castelló, Rafael Mayo, M. Mustafa Rafique, Bogdan Nicolae, Juan Carlos FernándezAbstract:Hybrid Cloud bursting (i.e., leasing temporary off-premise Cloud resources to boost the overall capacity during peak utilization) is a popular and cost-effective way to deal with the increasing complexity of big data analytics. It is particularly promising for iterative MapReduce applications that reuse massive amounts of input data at each iteration, which compensates for the high overhead and cost of concurrent data transfers from the on-premise to the off-premise VMs over a weak inter-site link that is of limited capacity. In this paper we study how to combine various MapReduce data locality techniques designed for Hybrid Cloud bursting in order to achieve scalability for iterative MapReduce applications in a cost-effective fashion. This is a non-trivial problem due to the complex interaction between the data movements over the weak link and the scheduling of computational tasks that have to adapt to the shifting data distribution. We show that using the right combination of techniques, iterative MapReduce applications can scale well in a Hybrid Cloud bursting scenario and come even close to the scalability observed in single sites.
Quan Zhang - One of the best experts on this subject based on the ideXlab platform.
-
firework data processing and sharing for Hybrid Cloud edge analytics
IEEE Transactions on Parallel and Distributed Systems, 2018Co-Authors: Quan Zhang, Weisong Shi, Qingyang Zhang, Hong ZhongAbstract:Now we are entering the era of the Internet of Everything (IoE) and billions of sensors and actuators are connected to the network. As one of the most sophisticated IoE applications, real-time video analytics is promising to significantly improve public safety, business intelligence, and healthcare & life science, among others. However, Cloud-centric video analytics requires that all video data must be preloaded to a centralized cluster or the Cloud, which suffers from high response latency and high cost of data transmission, given the scale of zettabytes of video data generated by IoE devices. Moreover, video data is rarely shared among multiple stakeholders due to various concerns, which restricts the practical deployment of video analytics that takes advantages of many data sources to make smart decisions. Furthermore, there is no efficient programming interface for developers and users to easily program and deploy IoE applications across geographically distributed computation resources. In this paper, we present a new computing framework, Firework , which facilitates distributed data processing and sharing for IoE applications via a virtual shared data view and service composition. We designed an easy-to-use programming interface for Firework to allow developers to program on Firework . This paper describes the system design, implementation, and programming interface of Firework . The experimental results of a video analytics application demonstrate that Firework reduces up to 19.52 percent of response latency and at least 72.77 percent of network bandwidth cost, compared to a Cloud-centric solution.
-
firework data analytics in Hybrid Cloud edge environment
2018Co-Authors: Jie Cao, Quan Zhang, Weisong ShiAbstract:Now we are entering the era of the Internet of Everything (IoE) moreover, billions of sensors and actuators are connected to the network.
