The Experts below are selected from a list of 3723 Experts worldwide ranked by ideXlab platform

Subrata Acharya - One of the best experts on this subject based on the ideXlab platform.

Jitka Komarkova - One of the best experts on this subject based on the ideXlab platform.

  • Processing of big educational data in the cloud using Apache Hadoop
    2016 International Conference on Information Society (i-Society), 2016
    Co-Authors: Renáta Máchová, Jitka Komarkova, Martin Lnenicka
    Abstract:

    The amount of available educational data is growing and the educational institutions try to organize and understand large complex datasets. Since these big data are an emerging field of study, there is no simple solution to process or analyze them. This paper aims to provide a model approach to process big educational data from the Moodle system in the cloud using the Apache Hadoop cluster. An experimental implementation of the proposed model approach is included to validate this solution. The results then suggest that the proposed model approach has a potential to be widely used for the processing of big educational data, especially with user interaction data.

  • ICCCI (2) - A Proposal of a Big Web Data Application and Archive for the Distributed Data Processing with Apache Hadoop
    Computational Collective Intelligence, 2015
    Co-Authors: Martin Lnenicka, Jan Hovad, Jitka Komarkova
    Abstract:

    In recent years, research on big data, data storage and other topics that represent innovations in the analytics field has become very popular. This paper describes a proposal of a big web data application and archive for the distributed data processing with Apache Hadoop, including the framework with selected methods, which can be used with this platform. It proposes a workflow to create a web content mining application and a big data archive, which uses modern technologies like Python, PHP, JavaScript, MySQL and cloud services. It also shows the overview about the architecture, methods and data structures used in the context of web mining, distributed processing and big data analytics.

  • a proposal of a big web data application and archive for the distributed data processing with Apache Hadoop
    International Conference on Computational Collective Intelligence, 2015
    Co-Authors: Martin Lnenicka, Jan Hovad, Jitka Komarkova
    Abstract:

    In recent years, research on big data, data storage and other topics that represent innovations in the analytics field has become very popular. This paper describes a proposal of a big web data application and archive for the distributed data processing with Apache Hadoop, including the framework with selected methods, which can be used with this platform. It proposes a workflow to create a web content mining application and a big data archive, which uses modern technologies like Python, PHP, JavaScript, MySQL and cloud services. It also shows the overview about the architecture, methods and data structures used in the context of web mining, distributed processing and big data analytics.

Lidong Chen - One of the best experts on this subject based on the ideXlab platform.

  • an approach for fast and parallel video processing on Apache Hadoop clusters
    International Conference on Multimedia and Expo, 2014
    Co-Authors: Hanlin Tan, Lidong Chen
    Abstract:

    This paper proposes an approach for fast and parallel video processing on MapReduce-based clusters such as Apache Hadoop. By utilizing clusters, the approach is able to handle large-scale of video data and the processing time can be significantly reduced. Technique details of performing video analysis on clusters are revealed, including method of porting typical video processing algorithms designed for a single computer to the proposed system. As case studies, face detection and motion detection and tracking algorithms have been implemented on clusters. Performance experiments on an Apache Hadoop cluster of six computers show that the system is able to reduce the running time of the two implemented algorithms to below 25% of that of a single computer. The applications of the system include smart city video surveillance, services provided by video sites and satellite image processing.

  • ICME - An approach for fast and parallel video processing on Apache Hadoop clusters
    2014 IEEE International Conference on Multimedia and Expo (ICME), 2014
    Co-Authors: Hanlin Tan, Lidong Chen
    Abstract:

    This paper proposes an approach for fast and parallel video processing on MapReduce-based clusters such as Apache Hadoop. By utilizing clusters, the approach is able to handle large-scale of video data and the processing time can be significantly reduced. Technique details of performing video analysis on clusters are revealed, including method of porting typical video processing algorithms designed for a single computer to the proposed system. As case studies, face detection and motion detection and tracking algorithms have been implemented on clusters. Performance experiments on an Apache Hadoop cluster of six computers show that the system is able to reduce the running time of the two implemented algorithms to below 25% of that of a single computer. The applications of the system include smart city video surveillance, services provided by video sites and satellite image processing.

Patricia Pitthan Barcelos - One of the best experts on this subject based on the ideXlab platform.

  • ICSA - Employment of Optimal Approximations on Apache Hadoop Checkpoint Technique for Performance Improvements
    2020 IEEE International Conference on Software Architecture (ICSA), 2020
    Co-Authors: Paulo Vinicius Cardoso, Rhauani Fazul, Patricia Pitthan Barcelos
    Abstract:

    The Checkpoint and Recovery (CR) technique is widely used due to its fault tolerance efficiency. The Apache Hadoop framework uses this technique as a way to avoid failures in its distributed file system. However, determining the optimal interval between successive checkpoints is a challenge, mainly inside Hadoop as it does not allow real-time modifications. The Dynamic Configuration Architecture (DCA) was created to solve this issue by enabling changes in the checkpoint period without any interruption of the Hadoop services. This paper presents improvements for the DCA through the configuration of the Hadoop checkpoint period in real-time based on optimal period approximations that were already endorsed by the literature. The proposed improvement depends on the tracking of the system resources. The data collected from these resources are stored in a history of attributes: a tree of monitored elements where data is updated as new observations are experienced in the system. This feature enables the user to estimate system factors so that our solution computes the checkpoints costs and the mean time between failures (MTBF). For the validation, experiments with transient failure in the NameNode were created and the usage of the history of attributes was tested in different scenarios. The evaluation results show that an adaptive configuration of checkpoint periods reduces the wasted time caused by failures in the NameNode and improves Hadoop performance. Also, the history of attributes demonstrated its value by providing an efficient way to estimate the system factors.

  • impacto do custo de salvamento e dos atributos de configuracao no checkpoint do Apache Hadoop
    Anais do XXXVII Simpósio Brasileiro de Redes de Computadores e Sistemas Distribuídos (SBRC 2019), 2019
    Co-Authors: Paulo Vinicius Cardoso, Patricia Pitthan Barcelos
    Abstract:

    The Apache Hadoop framework, which is used to process and store large amounts of data, uses the Checkpoint and Recovery technique to assist with failed recoveries of your distributed file system. However, efficient adaptations for time between Hadoop checkpoints depend on accurate system observations. The purpose of this paper is to estimate the cost of performing checkpoints and the average time between system failures from a history of observations. Factors are observed and analyzed for different variations of framework configuration and benchmark used.

  • performance evaluation of Apache Hadoop benchmarks under a dynamic checkpointing mechanism
    International Conference of the Chilean Computer Science Society, 2018
    Co-Authors: Paulo Vinicius Cardoso, Patricia Pitthan Barcelos
    Abstract:

    The increasingly robust High Performance Computing (HPC) systems have promoted many researches on fault tolerance mechanisms. However, the efficiency of these techniques is an important challenge, as the data and resources demand is also getting higher. in this work, we presents how the framework Apache Hadoop implements the Checkpoint and Recovery technique for fault tolerance providing on its distributed file system (Hadoop Distributed File System). The demand of large amounts of data from all applications supported by Hadoop framework can compromise the mechanism efficiency, once its configurations attributes are defined statically. So we present a dynamic configuration mechanism to checkpoint for Apache Hadoop through a resource monitoring, whose goal is to make the Hadoop checkpoint adaptable. The proposed mechanism is then evaluated and submitted to performance tests.

  • experimentation and analysis of dynamic checkpoint on Apache Hadoop with failure scenarios
    IEEE International Conference on High Performance Computing Data and Analytics, 2018
    Co-Authors: Paulo Vinicius Cardoso, Patricia Pitthan Barcelos
    Abstract:

    The growth of reliability problems on high performance systems has motivated searches for fault tolerance mechanisms. The Apache Hadoop framework, created to store and process large amounts of data, implements Checkpoint and Recovery to help on recovery process of its distributed file system (Hadoop Distributed File System - HDFS) in presence of failure. However, once configuration attributes can not be changed at runtime, bad choices may cause performance and reliability problems. This work uses a dynamic configuration mechanism for checkpoint on Hadoop and evaluates its performance on scenarios with induced fault on the master element of HDFS.

  • validation of a dynamic checkpoint mechanism for Apache Hadoop with failure scenarios
    2018 IEEE 19th Latin-American Test Symposium (LATS), 2018
    Co-Authors: Paulo Vinicius Cardoso, Patricia Pitthan Barcelos
    Abstract:

    New computational paradigms have created data intensive applications which have a demand for efficient and reliable processing platforms. High performance systems, used to answer this demand, have a increasing number of components such as nodes and cores. A multi component system may suffer with reliability and availability issues once the mean time between failures become smaller. Checkpoint and Recovery (CR) is a fault tolerance technique based on backward error recovery that focus on retrieving system safety state from backup saves. This paper shows the Checkpoint and Recovery technique implemented by Apache Hadoop, a framework that allows distributed processing of large datasets across clusters of computers. Hadoop uses the checkpoint technique to provides fault tolerance on Hadoop Distributed File System (HDFS). However, choosing an appropriate checkpoint interval is a major challenge once Hadoop defines the CR attributes statically. Then we propose a dynamic solution for checkpoint attributes configuration on HDFS, whose goal is to make it adaptable to system usage context. We expose a validation of both static and dynamic mechanisms on failure induced scenarios with DataNode crashes in order to determine the overhead of checkpoint and recovery steps.

Jason C. Cohen - One of the best experts on this subject based on the ideXlab platform.