The Experts below are selected from a list of 30654 Experts worldwide ranked by ideXlab platform

Francesco Pace - One of the best experts on this subject based on the ideXlab platform.

  • stocator providing high performance and fault tolerance for apache spark over Object Storage
    IEEE ACM International Symposium Cluster Cloud and Grid Computing, 2018
    Co-Authors: Gil Vernik, Michael Factor, Elliot K Kolodner, Pietro Michiardi, Effi Ofer, Francesco Pace
    Abstract:

    Until now Object Storage has not been a first-class citizen of the Apache Hadoop ecosystem including Apache Spark. Hadoop connectors to Object Storage have been based on file semantics, an impedance mismatch, which leads to low performance and the need for an additional consistent Storage system to achieve fault tolerance. In particular, Hadoop depends on its underlying Storage system and its associated connector for fault tolerance and allowing speculative execution. However, these characteristics are obtained through file operations that are not native for Object Storage, and are both costly and not atomic. As a result these connectors are not efficient and more importantly they cannot help with fault tolerance for Object Storage. We introduce Stocator, whose novel algorithm achieves both high performance and fault tolerance by taking advantage of Object Storage semantics. This greatly decreases the number of operations on Object Storage as well as enabling a much simpler approach to dealing with the eventually consistent semantics typical of Object Storage. We have implemented Stocator and shared it in open source. Performance testing with Apache Spark shows that it can be 18 times faster for write intensive workloads and can perform 30 times fewer operations on Object Storage than the legacy Hadoop connectors, reducing costs both for the client and the Object Storage service provider.

  • CCGrid - Stocator: providing high performance and fault tolerance for apache spark over Object Storage
    2018 18th IEEE ACM International Symposium on Cluster Cloud and Grid Computing (CCGRID), 2018
    Co-Authors: Gil Vernik, Michael Factor, Elliot K Kolodner, Pietro Michiardi, Effi Ofer, Francesco Pace
    Abstract:

    Until now Object Storage has not been a first-class citizen of the Apache Hadoop ecosystem including Apache Spark. Hadoop connectors to Object Storage have been based on file semantics, an impedance mismatch, which leads to low performance and the need for an additional consistent Storage system to achieve fault tolerance. In particular, Hadoop depends on its underlying Storage system and its associated connector for fault tolerance and allowing speculative execution. However, these characteristics are obtained through file operations that are not native for Object Storage, and are both costly and not atomic. As a result these connectors are not efficient and more importantly they cannot help with fault tolerance for Object Storage. We introduce Stocator, whose novel algorithm achieves both high performance and fault tolerance by taking advantage of Object Storage semantics. This greatly decreases the number of operations on Object Storage as well as enabling a much simpler approach to dealing with the eventually consistent semantics typical of Object Storage. We have implemented Stocator and shared it in open source. Performance testing with Apache Spark shows that it can be 18 times faster for write intensive workloads and can perform 30 times fewer operations on Object Storage than the legacy Hadoop connectors, reducing costs both for the client and the Object Storage service provider.

  • SYSTOR - Stocator: a high performance Object store connector for spark
    Proceedings of the 10th ACM International Systems and Storage Conference, 2017
    Co-Authors: Gil Vernik, Michael Factor, Elliot K Kolodner, Pietro Michiardi, Effi Ofer, Francesco Pace
    Abstract:

    Data is the natural resource of the 21st century. It is being produced at dizzying rates, e.g., for genomics by sequencers, for Media and Entertainment with very high resolution formats, and for Internet of Things (IoT) by multitudes of sensors. Object Stores such as AWS S3, Azure Blob Storage, and IBM Cloud Object Storage, are highly scalable distributed Storage systems that offer high capacity, cost effective Storage for this data. But it is not enough just to store data; we also need to derive value from it. Apache Spark is the leading big data analytics processing engine. It runs up to one hundred times faster than Hadoop MapReduce and combines SQL, streaming and complex analytics. In this poster we present Stocator, a high performance Storage connector, that enables Spark to work directly on data stored in Object Storage systems.

Yosef Moatti - One of the best experts on this subject based on the ideXlab platform.

  • Software-defined Object Storage in multi-tenant environments
    Future Generation Computer Systems, 2019
    Co-Authors: Raul Gracia-tinedo, Josep Sampé, Marc Sanchez-artigas, Pedro García-lópez, Gerard París, Yosef Moatti
    Abstract:

    Abstract Object stores are becoming pervasive due to their scalability and simplicityto manage data growth. Their rampant adoption, however, contrasts with their scant flexibility to support multi-tenancy. Very often, this results in a deficient adaptation of the system to the heterogeneous tenants’ demands and the multiple applications sharing the same Object Storage infrastructure. In this manuscript, we present Crystal, the first Software-Defined Storage (SDS) architecture whose core Objective is to efficiently support multi-tenancy in Object stores. Abide by the principles of software-defined systems, Crystal separates the Storage’s control logic from the underlying Storage nodes, proxies and caches, promoting the (logical) centralization of Storage control and the ability to program the Storage layer. For Storage control, Crystal enables high-level policies at the tenant, container, and Object granularities. It also translates these policies into a set of distributed controllers that can orchestrate computations at the Storage layer based on real-time workload information. We demonstrate Crystal through three use cases on top of OpenStack Swift: One that proves its Storage automation capabilities; one that exploits the pushdown of lambda functions on data streams to improve analytics data ingestion; and another that differentiates IO bandwidth in a multi-tenant scenario. We show that Crystal is an extensible platform to deploy new SDS services for Object stores with small overhead.

  • IOStack: Software-Defined Object Storage
    IEEE Internet Computing, 2016
    Co-Authors: Raul Gracia-tinedo, Josep Sampé, Marc Sanchez-artigas, Pedro García-lópez, Yosef Moatti, Eran Rom, Ramon Nou, Toni Cortes, Dalit Naor, William Oppermann
    Abstract:

    As the complexity and scale of cloud Storage systems grow, software-defined Storage (SDS) has become a prime candidate to simplify cloud Storage management. Here, the authors present IOStack, the first SDS architecture for Object stores (such as OpenStack Swift). At the control plane, the provisioning of SDS services to tenants is made according to a set of policies managed via a high-level domain-specific language (DSL). Policies can target Storage automation or specific service-level agreement (SLA) Objectives. At the data plane, policies define the enforcement of SDS services, namely filters, on a tenant's requests. Moreover, IOStack is a framework to build a variety of filters, ranging from general-purpose computations close to the data to specialized data management mechanisms. Experiments illustrate that IOStack enables easy and effective policy-based provisioning, which can significantly improve the operation of a multitenant Object store.

  • IOStack: Software-Defined Object Storage
    IEEE Internet Computing, 2016
    Co-Authors: Raul Gracia-tinedo, Josep Sampé, Marc Sanchez-artigas, Pedro García-lópez, Yosef Moatti, Eran Rom, Ramon Nou, Toni Cortes, Dalit Naor, William Oppermann
    Abstract:

    DOI: 10.1109/MIC.2016.46 URL: http://ieeexplore.ieee.org/document/7436653/ Filiació URV: SIThe complexity and scale of today¿s cloud Storage systems is growing fast. In response to these challenges, SoftwareDefined Storage (SDS) has recently become a prime candidate to simplify Storage management in the cloud. This article presents IOStack: The first SDS architecture for Object stores (OpenStack Swift). At the control plane, the provisioning of SDS services to tenants is made according to a set of policies managed via a high-level DSL. Policies may target Storage automation and/or specific SLA Objectives. At the data plane, policies define the enforcement of SDS services, namely filters, on a tenant¿s requests. Moreover, IOStack is a framework to build a variety of filters, ranging from general-purpose computations close to the data to specialized data management mechanisms. Our experiments illustrate that IOStack enables easy and effective policy-based provisioning, which can significantly improve the operation of a multi-tenant Object store

Gil Vernik - One of the best experts on this subject based on the ideXlab platform.

  • stocator providing high performance and fault tolerance for apache spark over Object Storage
    IEEE ACM International Symposium Cluster Cloud and Grid Computing, 2018
    Co-Authors: Gil Vernik, Michael Factor, Elliot K Kolodner, Pietro Michiardi, Effi Ofer, Francesco Pace
    Abstract:

    Until now Object Storage has not been a first-class citizen of the Apache Hadoop ecosystem including Apache Spark. Hadoop connectors to Object Storage have been based on file semantics, an impedance mismatch, which leads to low performance and the need for an additional consistent Storage system to achieve fault tolerance. In particular, Hadoop depends on its underlying Storage system and its associated connector for fault tolerance and allowing speculative execution. However, these characteristics are obtained through file operations that are not native for Object Storage, and are both costly and not atomic. As a result these connectors are not efficient and more importantly they cannot help with fault tolerance for Object Storage. We introduce Stocator, whose novel algorithm achieves both high performance and fault tolerance by taking advantage of Object Storage semantics. This greatly decreases the number of operations on Object Storage as well as enabling a much simpler approach to dealing with the eventually consistent semantics typical of Object Storage. We have implemented Stocator and shared it in open source. Performance testing with Apache Spark shows that it can be 18 times faster for write intensive workloads and can perform 30 times fewer operations on Object Storage than the legacy Hadoop connectors, reducing costs both for the client and the Object Storage service provider.

  • CCGrid - Stocator: providing high performance and fault tolerance for apache spark over Object Storage
    2018 18th IEEE ACM International Symposium on Cluster Cloud and Grid Computing (CCGRID), 2018
    Co-Authors: Gil Vernik, Michael Factor, Elliot K Kolodner, Pietro Michiardi, Effi Ofer, Francesco Pace
    Abstract:

    Until now Object Storage has not been a first-class citizen of the Apache Hadoop ecosystem including Apache Spark. Hadoop connectors to Object Storage have been based on file semantics, an impedance mismatch, which leads to low performance and the need for an additional consistent Storage system to achieve fault tolerance. In particular, Hadoop depends on its underlying Storage system and its associated connector for fault tolerance and allowing speculative execution. However, these characteristics are obtained through file operations that are not native for Object Storage, and are both costly and not atomic. As a result these connectors are not efficient and more importantly they cannot help with fault tolerance for Object Storage. We introduce Stocator, whose novel algorithm achieves both high performance and fault tolerance by taking advantage of Object Storage semantics. This greatly decreases the number of operations on Object Storage as well as enabling a much simpler approach to dealing with the eventually consistent semantics typical of Object Storage. We have implemented Stocator and shared it in open source. Performance testing with Apache Spark shows that it can be 18 times faster for write intensive workloads and can perform 30 times fewer operations on Object Storage than the legacy Hadoop connectors, reducing costs both for the client and the Object Storage service provider.

  • SYSTOR - Stocator: a high performance Object store connector for spark
    Proceedings of the 10th ACM International Systems and Storage Conference, 2017
    Co-Authors: Gil Vernik, Michael Factor, Elliot K Kolodner, Pietro Michiardi, Effi Ofer, Francesco Pace
    Abstract:

    Data is the natural resource of the 21st century. It is being produced at dizzying rates, e.g., for genomics by sequencers, for Media and Entertainment with very high resolution formats, and for Internet of Things (IoT) by multitudes of sensors. Object Stores such as AWS S3, Azure Blob Storage, and IBM Cloud Object Storage, are highly scalable distributed Storage systems that offer high capacity, cost effective Storage for this data. But it is not enough just to store data; we also need to derive value from it. Apache Spark is the leading big data analytics processing engine. It runs up to one hundred times faster than Hadoop MapReduce and combines SQL, streaming and complex analytics. In this poster we present Stocator, a high performance Storage connector, that enables Spark to work directly on data stored in Object Storage systems.

Michael J Freedman - One of the best experts on this subject based on the ideXlab platform.

  • Object Storage on craq high throughput chain replication for read mostly workloads
    USENIX Annual Technical Conference, 2009
    Co-Authors: Jeff Terrace, Michael J Freedman
    Abstract:

    Massive Storage systems typically replicate and partition data over many potentially-faulty components to provide both reliability and scalability. Yet many commerciallydeployed systems, especially those designed for interactive use by customers, sacrifice stronger consistency properties in the desire for greater availability and higher throughput. This paper describes the design, implementation, and evaluation of CRAQ, a distributed Object-Storage system that challenges this inflexible tradeoff. Our basic approach, an improvement on Chain Replication, maintains strong consistency while greatly improving read throughput. By distributing load across all Object replicas, CRAQ scales linearly with chain size without increasing consistency coordination. At the same time, it exposes noncommitted operations for weaker consistency guarantees when this suffices for some applications, which is especially useful under periods of high system churn. This paper explores additional design and implementation considerations for geo-replicated CRAQ Storage across multiple datacenters to provide locality-optimized operations. We also discuss multi-Object atomic updates and multicast optimizations for large-Object updates.

  • USENIX Annual Technical Conference - Object Storage on CRAQ: high-throughput chain replication for read-mostly workloads
    2009
    Co-Authors: Jeff Terrace, Michael J Freedman
    Abstract:

    Massive Storage systems typically replicate and partition data over many potentially-faulty components to provide both reliability and scalability. Yet many commerciallydeployed systems, especially those designed for interactive use by customers, sacrifice stronger consistency properties in the desire for greater availability and higher throughput. This paper describes the design, implementation, and evaluation of CRAQ, a distributed Object-Storage system that challenges this inflexible tradeoff. Our basic approach, an improvement on Chain Replication, maintains strong consistency while greatly improving read throughput. By distributing load across all Object replicas, CRAQ scales linearly with chain size without increasing consistency coordination. At the same time, it exposes noncommitted operations for weaker consistency guarantees when this suffices for some applications, which is especially useful under periods of high system churn. This paper explores additional design and implementation considerations for geo-replicated CRAQ Storage across multiple datacenters to provide locality-optimized operations. We also discuss multi-Object atomic updates and multicast optimizations for large-Object updates.

Raul Gracia-tinedo - One of the best experts on this subject based on the ideXlab platform.

  • Software-defined Object Storage in multi-tenant environments
    Future Generation Computer Systems, 2019
    Co-Authors: Raul Gracia-tinedo, Josep Sampé, Marc Sanchez-artigas, Pedro García-lópez, Gerard París, Yosef Moatti
    Abstract:

    Abstract Object stores are becoming pervasive due to their scalability and simplicityto manage data growth. Their rampant adoption, however, contrasts with their scant flexibility to support multi-tenancy. Very often, this results in a deficient adaptation of the system to the heterogeneous tenants’ demands and the multiple applications sharing the same Object Storage infrastructure. In this manuscript, we present Crystal, the first Software-Defined Storage (SDS) architecture whose core Objective is to efficiently support multi-tenancy in Object stores. Abide by the principles of software-defined systems, Crystal separates the Storage’s control logic from the underlying Storage nodes, proxies and caches, promoting the (logical) centralization of Storage control and the ability to program the Storage layer. For Storage control, Crystal enables high-level policies at the tenant, container, and Object granularities. It also translates these policies into a set of distributed controllers that can orchestrate computations at the Storage layer based on real-time workload information. We demonstrate Crystal through three use cases on top of OpenStack Swift: One that proves its Storage automation capabilities; one that exploits the pushdown of lambda functions on data streams to improve analytics data ingestion; and another that differentiates IO bandwidth in a multi-tenant scenario. We show that Crystal is an extensible platform to deploy new SDS services for Object stores with small overhead.

  • IOStack: Software-Defined Object Storage
    IEEE Internet Computing, 2016
    Co-Authors: Raul Gracia-tinedo, Josep Sampé, Marc Sanchez-artigas, Pedro García-lópez, Yosef Moatti, Eran Rom, Ramon Nou, Toni Cortes, Dalit Naor, William Oppermann
    Abstract:

    As the complexity and scale of cloud Storage systems grow, software-defined Storage (SDS) has become a prime candidate to simplify cloud Storage management. Here, the authors present IOStack, the first SDS architecture for Object stores (such as OpenStack Swift). At the control plane, the provisioning of SDS services to tenants is made according to a set of policies managed via a high-level domain-specific language (DSL). Policies can target Storage automation or specific service-level agreement (SLA) Objectives. At the data plane, policies define the enforcement of SDS services, namely filters, on a tenant's requests. Moreover, IOStack is a framework to build a variety of filters, ranging from general-purpose computations close to the data to specialized data management mechanisms. Experiments illustrate that IOStack enables easy and effective policy-based provisioning, which can significantly improve the operation of a multitenant Object store.

  • IOStack: Software-Defined Object Storage
    IEEE Internet Computing, 2016
    Co-Authors: Raul Gracia-tinedo, Josep Sampé, Marc Sanchez-artigas, Pedro García-lópez, Yosef Moatti, Eran Rom, Ramon Nou, Toni Cortes, Dalit Naor, William Oppermann
    Abstract:

    DOI: 10.1109/MIC.2016.46 URL: http://ieeexplore.ieee.org/document/7436653/ Filiació URV: SIThe complexity and scale of today¿s cloud Storage systems is growing fast. In response to these challenges, SoftwareDefined Storage (SDS) has recently become a prime candidate to simplify Storage management in the cloud. This article presents IOStack: The first SDS architecture for Object stores (OpenStack Swift). At the control plane, the provisioning of SDS services to tenants is made according to a set of policies managed via a high-level DSL. Policies may target Storage automation and/or specific SLA Objectives. At the data plane, policies define the enforcement of SDS services, namely filters, on a tenant¿s requests. Moreover, IOStack is a framework to build a variety of filters, ranging from general-purpose computations close to the data to specialized data management mechanisms. Our experiments illustrate that IOStack enables easy and effective policy-based provisioning, which can significantly improve the operation of a multi-tenant Object store