The Experts below are selected from a list of 189 Experts worldwide ranked by ideXlab platform

Stephanie Pamela Adithama - One of the best experts on this subject based on the ideXlab platform.

  • rancang bangun real time business intelligence untuk subjek kegiatan akademik pada universitas menggunakan Change Data Capture
    Journal of Biomedical Informatics, 2014
    Co-Authors: Stephanie Pamela Adithama
    Abstract:

    Abstract. The running of academic activities in university continuously adds more Data to the existing operational system. The Data are not ready for the university strategic decision making, preparing reports for accreditation purposes and academic units. Real-time business intelligence application using Data warehouse can become a solution for Data analysis. The process of creating a Data warehouse includes designing Data warehouse, retrieving academic Data from multiple Data sources, extracting, transforming, loading (ETL) process, creating cube; and generating report. ETL processes are conducted by using a Pull Change Data Capture approach so that Data Changes during a certain period can be transferred in real-time. The higher the frequency of Data Change requests brings us closer to real-time and requires less time than loading all the Data. Keywords: real-time, business intelligence, Data warehouse, academic, Change Data Capture   Abstrak.  Kegiatan akademik di universitas berjalan terus menerus dan semakin menambah banyak Data pada sistem operasional yang sudah ada. Data tersebut masih belum dapat dimanfaatkan oleh pihak universitas dalam pengambilan keputusan strategis, pembuatan laporan untuk keperluan akreditasi dan unit-unit akademik. Aplikasi real-time business intelligence menggunakan Data warehouse menjadi solusi untuk analisa Data. Proses pembuatan Data warehouse meliputi perancangan Data warehouse; pengambilan Data akademik dari sumber Data; proses extraction, transformation, loading (ETL); pembuatan cube; dan pembuatan laporan. Proses ETL dilakukan menggunakan pendekatan Change Data Capture Pull agar perubahan Data selama periode tertentu dapat dipindahkan secara real-time. Semakin tinggi frekuensi permintaan perubahan Data akan semakin mendekati real-time dan semakin membutuhkan waktu yang singkat dibandingkan dengan me-load semua Data. Kata Kunci: real-time, business intelligence, Data warehouse, akademik, Change Data Capture

  • PENGEMBANGAN APLIKASI REAL-TIME BUSINESS INTELLIGENCE BERBASIS WEB UNTUK SUBJEK KEGIATAN AKADEMIK PADA UNIVERSITAS
    2013
    Co-Authors: Stephanie Pamela Adithama
    Abstract:

    Academic activities at Universitas Atma Jaya Yogyakarta (UAJY) runs continuously and add more Data to the existing operational system. This Data still can not be used by the university for strategic decision making. Currently, decisions are made based on habits and general logic that not based on a comprehensive Data and facts. Meanwhile, preparing reports required for accreditation purposes and preparing reports on academic units. Real-time business intelligence (BI) application using Data warehouse to be solution for Data analysis. The process of creating a Data warehouse includes design of Data warehouse; retrieval of academic Data from multiple Data sources; extraction, transformation, loading (ETL) process; cube creation; and report generation. ETL processes are conducted using Pull Change Data Capture approach so that Data Changes during a certain period can be transferred in realtime. The tools used are Microsoft Visual Studio 2008, SQL Server 2008 and Report Portal 4.0. With the implementation of this application, is expected to meet the university management needs to know information about the academic activities, assist in reporting, and help make decisions more effectively and efficiently compared to just using intuition.

Bo Yang - One of the best experts on this subject based on the ideXlab platform.

  • 3PGCIC - Live Data Replication Approach from Relational Tables to Schema-Free Collections Using Stream Processing Framework
    2015 10th International Conference on P2P Parallel Grid Cloud and Internet Computing (3PGCIC), 2015
    Co-Authors: Kun Ma, Bo Yang
    Abstract:

    Recent researches focus on the Data replication issue from relational tables to schema-free collections in a batch processing way. However, there are few publications on live Data replication in real time. In this paper, we attempt to address this legacy issue with new stream processing framework. The process of replication consists of log-based Change Data Capture and stream-based Data replication. Data replication mappings are present, and the proposed architecture of stream processing framework including column grouping, column merging and column versioning, is introduced to avoid Data lost in case of failure. Finally, our experimental evaluation of live Data replication approach with stream processing framework shows the higher effectiveness and efficiency than current methods.

  • Log-based Change Data Capture from schema-free document stores using MapReduce
    2015 International Conference on Cloud Technologies and Applications (CloudTech), 2015
    Co-Authors: Bo Yang
    Abstract:

    Change Data Capture (CDC) is an approach to Data integration that is used to determine and track the Data that has Changed so that action can be taken using the Change Data. However, the state of art of Change Data Capture (CDC) in the context of document-oriented NoSQL Databases is not mature. Therefore, it is urgent to require a NoSQL CDC solution. Although some manufacturers of NoSQL Databases start to research on CDC for NoSQL, these approaches are just for the specific product. In our paper, we propose a log-based CDC approach from abstract schema-free document stores using MapReduce. The process is divided into map and reduce procedures, benefited from MapReduce framework, to generate cell state models (CSMs). In order to infinitely look back to any revision, we enable our proposed CSM to support copy-modify-merge model to manage the revisions of Change Data. Finally, experimental results show that this approach is independent and appropriate for document stores, with high performance and throughput capacity.

  • Live Data Replication Approach from Relational Tables to Schema-Free Collections Using Stream Processing Framework
    2015 10th International Conference on P2P Parallel Grid Cloud and Internet Computing (3PGCIC), 2015
    Co-Authors: Bo Yang
    Abstract:

    Recent researches focus on the Data replication issue from relational tables to schema-free collections in a batch processing way. However, there are few publications on live Data replication in real time. In this paper, we attempt to address this legacy issue with new stream processing framework. The process of replication consists of log-based Change Data Capture and stream-based Data replication. Data replication mappings are present, and the proposed architecture of stream processing framework including column grouping, column merging and column versioning, is introduced to avoid Data lost in case of failure. Finally, our experimental evaluation of live Data replication approach with stream processing framework shows the higher effectiveness and efficiency than current methods.

Anthony J. Beaumont - One of the best experts on this subject based on the ideXlab platform.

  • True Real-Time Change Data Capture with Web Service Database Encapsulation
    2010 6th World Congress on Services, 2010
    Co-Authors: Mitchell J. Eccles, David J. Evans, Anthony J. Beaumont
    Abstract:

    This research is investigating the claim that Change Data Capture (CDC) technologies Capture Data Changes in real-time. Based on theory, our hypothesis states that real-time CDC is not achievable with traditional approaches (log scanning, triggers and timestamps). Traditional approaches to CDC require a resource to be polled, which prevents true real-time CDC. We propose an approach to CDC that encapsulates the Data source with a set of web services. These web services will propagate the Changes to the targets and eliminate the need for polling. Additionally we propose a framework for CDC technologies that allow Changes to flow from source to target. This paper discusses current CDC technologies and presents the theory about why they are unable to deliver Changes in real-time. Following, we discuss our web service approach to CDC and accompanying framework, explaining how they can produce real-time CDC. The paper concludes with a discussion on the research required to investigate the real-time capabilities of CDC technologies.

  • SERVICES - True Real-Time Change Data Capture with Web Service Database Encapsulation
    2010 6th World Congress on Services, 2010
    Co-Authors: Mitchell J. Eccles, David J. Evans, Anthony J. Beaumont
    Abstract:

    This research is investigating the claim that Change Data Capture (CDC) technologies Capture Data Changes in real-time. Based on theory, our hypothesis states that real-time CDC is not achievable with traditional approaches (log scanning, triggers and timestamps). Traditional approaches to CDC require a resource to be polled, which prevents true real-time CDC. We propose an approach to CDC that encapsulates the Data source with a set of web services. These web services will propagate the Changes to the targets and eliminate the need for polling. Additionally we propose a framework for CDC technologies that allow Changes to flow from source to target. This paper discusses current CDC technologies and presents the theory about why they are unable to deliver Changes in real-time. Following, we discuss our web service approach to CDC and accompanying framework, explaining how they can produce real-time CDC. The paper concludes with a discussion on the research required to investigate the real-time capabilities of CDC technologies.

Yong Hu - One of the best experts on this subject based on the ideXlab platform.

  • Efficiently Extracting Change Data from Column Oriented NoSQL Databases
    Advances in Intelligent Systems and Applications - Volume 2, 2020
    Co-Authors: Yong Hu, Weiping Qu
    Abstract:

    This paper explores the appropriate Change Data Capture approaches implemented by exploiting the MapReduce framework in the context of column- oriented NoSQL Databases. Change Data Capture describes how the system acquires Change Data from Data sources. It is the most significant preparatory task for the incremental re-computation. Based on analyzing the core features of column-oriented NoSQL Databases, we define a notion of net-effect Change Data and propose three Change Data Capture approaches, namely, Snapshot Differential, Log-based approach and Trigger-based approach (Change Tracking Table).

  • ADBIS (Short Papers and Workshops) - Incrementally Maintaining Materialized Temporal Views in Column-Oriented NoSQL Databases with Partial Deltas
    Communications in Computer and Information Science, 2015
    Co-Authors: Yong Hu, Stefan Deßloch
    Abstract:

    Different from the relational Database systems, each column in a column-oriented NoSQL Database (CoNoSQLDB) maintains multiple Data versions in which each Data version is attached with an explicit timestamp (TS). In this paper, we study how to maintain the materialized temporal views (MTVs) in CoNoSQLDBs with partial temporal Data-Changes (deltas). We first review our previous work and indicate that not all Change-Data Capture (CDC) approaches are able to provide complete deltas. Then, we propose approaches for maintaining MTVs with partial deltas.

  • Change Data Capture in NoSQL Databases: A functional and performance comparison
    2015 IEEE Symposium on Computers and Communication (ISCC), 2015
    Co-Authors: Felipe Mathias Schmidt, Claudio Geyer, Alberto Schaeffer-filho, Stefan Debloch, Yong Hu
    Abstract:

    Requirements for Data storage and processing have reached new levels, with applications relying on the analysis of large amounts of Data in order to support everyday life services to end users. Since the costs of maintaining and managing Databases are significant, Change Data Capture (CDC) techniques can be used to determine which parts of a Data source have Changed, and thus assist in the management of large volumes of Data in Data warehouses. In this paper we investigate a number of CDC techniques suitable for NoSQL Databases. CDC techniques can be used to track modifications in a source Database, which later can be made available to a target Database. Our base system and testbed are based on Apache Cassandra, which is a NoSQL Database that offers high performance and scalability. Cassandra is combined with a MapReduce framework, which is used to implement the logic of each CDC technique and is suitable for highly distributed and parallel computing. This paper also presents both a functional comparison of the different CDC techniques, as well as a performance evaluation in a real testbed.

  • ISCC - Change Data Capture in NoSQL Databases: A functional and performance comparison
    2015 IEEE Symposium on Computers and Communication (ISCC), 2015
    Co-Authors: Felipe Mathias Schmidt, Alberto Schaeffer-filho, Stefan Debloch, Cláudio F. R. Geyer, Yong Hu
    Abstract:

    Requirements for Data storage and processing have reached new levels, with applications relying on the analysis of large amounts of Data in order to support everyday life services to end users. Since the costs of maintaining and managing Databases are significant, Change Data Capture (CDC) techniques can be used to determine which parts of a Data source have Changed, and thus assist in the management of large volumes of Data in Data warehouses. In this paper we investigate a number of CDC techniques suitable for NoSQL Databases. CDC techniques can be used to track modifications in a source Database, which later can be made available to a target Database. Our base system and testbed are based on Apache Cassandra, which is a NoSQL Database that offers high performance and scalability. Cassandra is combined with a MapReduce framework, which is used to implement the logic of each CDC technique and is suitable for highly distributed and parallel computing. This paper also presents both a functional comparison of the different CDC techniques, as well as a performance evaluation in a real testbed.

  • Extracting deltas from column oriented NoSQL Databases for different incremental applications and diverse Data targets
    Data & Knowledge Engineering, 2014
    Co-Authors: Yong Hu, Stefan Deßloch
    Abstract:

    This paper describes the Change Data Capture (CDC) problems in the context of column-oriented NoSQL Databases (CoNoSQLDBs). CDC is a term mostly used by ETL tools and Data warehousing environments (DW) to depict a Data processing of extracting Data Changes made at the Data sources. Based on analyzing the impacts and constraints caused by the core features of CoNoSQLDBs, we propose a logical Change Data (delta) model and the corresponding delta representations which could work with different incremental applications and diverse Data targets. Moreover, we present five feasible CDC approaches, i.e. Timestamp-based approach, Audit-column approach, Log-based approach, Trigger-based approach and Snapshot differential approach and indicate the performance winners under different circumstances.

Mitchell J. Eccles - One of the best experts on this subject based on the ideXlab platform.

  • True Real-Time Change Data Capture with Web Service Database Encapsulation
    2010 6th World Congress on Services, 2010
    Co-Authors: Mitchell J. Eccles, David J. Evans, Anthony J. Beaumont
    Abstract:

    This research is investigating the claim that Change Data Capture (CDC) technologies Capture Data Changes in real-time. Based on theory, our hypothesis states that real-time CDC is not achievable with traditional approaches (log scanning, triggers and timestamps). Traditional approaches to CDC require a resource to be polled, which prevents true real-time CDC. We propose an approach to CDC that encapsulates the Data source with a set of web services. These web services will propagate the Changes to the targets and eliminate the need for polling. Additionally we propose a framework for CDC technologies that allow Changes to flow from source to target. This paper discusses current CDC technologies and presents the theory about why they are unable to deliver Changes in real-time. Following, we discuss our web service approach to CDC and accompanying framework, explaining how they can produce real-time CDC. The paper concludes with a discussion on the research required to investigate the real-time capabilities of CDC technologies.

  • SERVICES - True Real-Time Change Data Capture with Web Service Database Encapsulation
    2010 6th World Congress on Services, 2010
    Co-Authors: Mitchell J. Eccles, David J. Evans, Anthony J. Beaumont
    Abstract:

    This research is investigating the claim that Change Data Capture (CDC) technologies Capture Data Changes in real-time. Based on theory, our hypothesis states that real-time CDC is not achievable with traditional approaches (log scanning, triggers and timestamps). Traditional approaches to CDC require a resource to be polled, which prevents true real-time CDC. We propose an approach to CDC that encapsulates the Data source with a set of web services. These web services will propagate the Changes to the targets and eliminate the need for polling. Additionally we propose a framework for CDC technologies that allow Changes to flow from source to target. This paper discusses current CDC technologies and presents the theory about why they are unable to deliver Changes in real-time. Following, we discuss our web service approach to CDC and accompanying framework, explaining how they can produce real-time CDC. The paper concludes with a discussion on the research required to investigate the real-time capabilities of CDC technologies.