The Experts below are selected from a list of 18 Experts worldwide ranked by ideXlab platform

Guilan Wang - One of the best experts on this subject based on the ideXlab platform.

  • Cache Conscious Star-Join in MapReduce Environments
    2015
    Co-Authors: Guoliang Zhou, North China, Electric Power, Yongli Zhu, Guilan Wang
    Abstract:

    With the popularity of big data and cloud computing, data parallel framework MapReduce based data warehouse sys-tems are used widely. Column store is a default data place-ment in these systems. Traditionally star join is a core operation in the data warehouse. However, little related work study star join in Column store and MapReduce envi-ronments. This paper proposes two new cache conscious algorithms Multi-Fragment-Replication Join (MFRJ) and MapReduce-Invisible Join (MRIJ) in MapReduce environ-ments. All these algorithms avoid fact table data movement and are cache conscious in each MapReduce node. In ad-dition, fact table is partitioned into several Column groups for cache optimization in MFRJ; One group contains all of Foreign Key Columns and each measure Column is a group. In MRIJ, each Column is separately processed one by one which has higher cache utilization and avoids frequently cache miss from one Column to the other Column. MRIJ is composed of several map operation on dimension tables and one MapRe-duce job. We also apply MRIJ on RCFile in Hive. All oper-ations are processed in mapping phase and avoid high cost of shuffle and reduce operation. If the dimension tables are big enough and cannot cache in local memory, MRIJ is di-vided into two phases, firstly each dimension table join with corresponding Foreign Key Column in fact table as commonly map reduce join concurrently or serially; secondly all inter-nal results joined for final results based on position index. This strategy also can be applied to other multi-table join. In order to reduce network I/O, dimension table and the fact table Foreign Key Column are co-location storage. Our experimental results in cluster environments show that our algorithms outperform existing approaches in Hive system. Categories and Subject Descriptor

  • Cloud-I - Cache conscious star-join in MapReduce environments
    Proceedings of the 2nd International Workshop on Cloud Intelligence - Cloud-I '13, 2013
    Co-Authors: Guoliang Zhou, Guilan Wang
    Abstract:

    With the popularity of big data and cloud computing, data parallel framework MapReduce based data warehouse systems are used widely. Column store is a default data placement in these systems. Traditionally star join is a core operation in the data warehouse. However, little related work study star join in Column store and MapReduce environments. This paper proposes two new cache conscious algorithms Multi-Fragment-Replication Join (MFRJ) and MapReduce-Invisible Join (MRIJ) in MapReduce environments. All these algorithms avoid fact table data movement and are cache conscious in each MapReduce node. In addition, fact table is partitioned into several Column groups for cache optimization in MFRJ; One group contains all of Foreign Key Columns and each measure Column is a group. In MRIJ, each Column is separately processed one by one which has higher cache utilization and avoids frequently cache miss from one Column to the other Column. MRIJ is composed of several map operation on dimension tables and one MapReduce job. We also apply MRIJ on RCFile in Hive. All operations are processed in mapping phase and avoid high cost of shuffle and reduce operation. If the dimension tables are big enough and cannot cache in local memory, MRIJ is divided into two phases, firstly each dimension table join with corresponding Foreign Key Column in fact table as commonly map reduce join concurrently or serially; secondly all internal results joined for final results based on position index. This strategy also can be applied to other multi-table join. In order to reduce network I/O, dimension table and the fact table Foreign Key Column are co-location storage. Our experimental results in cluster environments show that our algorithms outperform existing approaches in Hive system.

Guoliang Zhou - One of the best experts on this subject based on the ideXlab platform.

  • Cache Conscious Star-Join in MapReduce Environments
    2015
    Co-Authors: Guoliang Zhou, North China, Electric Power, Yongli Zhu, Guilan Wang
    Abstract:

    With the popularity of big data and cloud computing, data parallel framework MapReduce based data warehouse sys-tems are used widely. Column store is a default data place-ment in these systems. Traditionally star join is a core operation in the data warehouse. However, little related work study star join in Column store and MapReduce envi-ronments. This paper proposes two new cache conscious algorithms Multi-Fragment-Replication Join (MFRJ) and MapReduce-Invisible Join (MRIJ) in MapReduce environ-ments. All these algorithms avoid fact table data movement and are cache conscious in each MapReduce node. In ad-dition, fact table is partitioned into several Column groups for cache optimization in MFRJ; One group contains all of Foreign Key Columns and each measure Column is a group. In MRIJ, each Column is separately processed one by one which has higher cache utilization and avoids frequently cache miss from one Column to the other Column. MRIJ is composed of several map operation on dimension tables and one MapRe-duce job. We also apply MRIJ on RCFile in Hive. All oper-ations are processed in mapping phase and avoid high cost of shuffle and reduce operation. If the dimension tables are big enough and cannot cache in local memory, MRIJ is di-vided into two phases, firstly each dimension table join with corresponding Foreign Key Column in fact table as commonly map reduce join concurrently or serially; secondly all inter-nal results joined for final results based on position index. This strategy also can be applied to other multi-table join. In order to reduce network I/O, dimension table and the fact table Foreign Key Column are co-location storage. Our experimental results in cluster environments show that our algorithms outperform existing approaches in Hive system. Categories and Subject Descriptor

  • Cloud-I - Cache conscious star-join in MapReduce environments
    Proceedings of the 2nd International Workshop on Cloud Intelligence - Cloud-I '13, 2013
    Co-Authors: Guoliang Zhou, Guilan Wang
    Abstract:

    With the popularity of big data and cloud computing, data parallel framework MapReduce based data warehouse systems are used widely. Column store is a default data placement in these systems. Traditionally star join is a core operation in the data warehouse. However, little related work study star join in Column store and MapReduce environments. This paper proposes two new cache conscious algorithms Multi-Fragment-Replication Join (MFRJ) and MapReduce-Invisible Join (MRIJ) in MapReduce environments. All these algorithms avoid fact table data movement and are cache conscious in each MapReduce node. In addition, fact table is partitioned into several Column groups for cache optimization in MFRJ; One group contains all of Foreign Key Columns and each measure Column is a group. In MRIJ, each Column is separately processed one by one which has higher cache utilization and avoids frequently cache miss from one Column to the other Column. MRIJ is composed of several map operation on dimension tables and one MapReduce job. We also apply MRIJ on RCFile in Hive. All operations are processed in mapping phase and avoid high cost of shuffle and reduce operation. If the dimension tables are big enough and cannot cache in local memory, MRIJ is divided into two phases, firstly each dimension table join with corresponding Foreign Key Column in fact table as commonly map reduce join concurrently or serially; secondly all internal results joined for final results based on position index. This strategy also can be applied to other multi-table join. In order to reduce network I/O, dimension table and the fact table Foreign Key Column are co-location storage. Our experimental results in cluster environments show that our algorithms outperform existing approaches in Hive system.

Joe Celko - One of the best experts on this subject based on the ideXlab platform.

  • Chapter 6 – Keys
    Joe Celko's Data Measurements and Standards in SQL, 2010
    Co-Authors: Joe Celko
    Abstract:

    Publisher Summary This chapter focuses on the Keys used in a database. A Key by definition is unique and cannot be null. The surrogate Key is one of the most misunderstood concepts in relational database management systems (RDBMS), where its newer users fake surrogate Keys with various proprietary autonumbering features in their database product. Autonumbering is an exposed physical locator and not a surrogate. Because a surrogate Key acts like an index, it is created by the user, managed by the system, and never seen by a user, and thus it is never used in queries, DRI, or anything else that a user does. One of the two arguments usually made for machine-generated physical locators (“fake pointers”) is that a multiColumn Key is so lengthy that performance is adversely affected. However, this argument only applies to certain implementations such as the Foreign Key Column(s) in the SQL Server family and the pointers used by Sybase SQL anywhere. The second argument for fake pointers is that a suitable natural Key does not exist. However, a natural Key has to exist according to the Law of Identity, although it might not be a suitable Key due to its bad design.

North China - One of the best experts on this subject based on the ideXlab platform.

  • Cache Conscious Star-Join in MapReduce Environments
    2015
    Co-Authors: Guoliang Zhou, North China, Electric Power, Yongli Zhu, Guilan Wang
    Abstract:

    With the popularity of big data and cloud computing, data parallel framework MapReduce based data warehouse sys-tems are used widely. Column store is a default data place-ment in these systems. Traditionally star join is a core operation in the data warehouse. However, little related work study star join in Column store and MapReduce envi-ronments. This paper proposes two new cache conscious algorithms Multi-Fragment-Replication Join (MFRJ) and MapReduce-Invisible Join (MRIJ) in MapReduce environ-ments. All these algorithms avoid fact table data movement and are cache conscious in each MapReduce node. In ad-dition, fact table is partitioned into several Column groups for cache optimization in MFRJ; One group contains all of Foreign Key Columns and each measure Column is a group. In MRIJ, each Column is separately processed one by one which has higher cache utilization and avoids frequently cache miss from one Column to the other Column. MRIJ is composed of several map operation on dimension tables and one MapRe-duce job. We also apply MRIJ on RCFile in Hive. All oper-ations are processed in mapping phase and avoid high cost of shuffle and reduce operation. If the dimension tables are big enough and cannot cache in local memory, MRIJ is di-vided into two phases, firstly each dimension table join with corresponding Foreign Key Column in fact table as commonly map reduce join concurrently or serially; secondly all inter-nal results joined for final results based on position index. This strategy also can be applied to other multi-table join. In order to reduce network I/O, dimension table and the fact table Foreign Key Column are co-location storage. Our experimental results in cluster environments show that our algorithms outperform existing approaches in Hive system. Categories and Subject Descriptor

Electric Power - One of the best experts on this subject based on the ideXlab platform.

  • Cache Conscious Star-Join in MapReduce Environments
    2015
    Co-Authors: Guoliang Zhou, North China, Electric Power, Yongli Zhu, Guilan Wang
    Abstract:

    With the popularity of big data and cloud computing, data parallel framework MapReduce based data warehouse sys-tems are used widely. Column store is a default data place-ment in these systems. Traditionally star join is a core operation in the data warehouse. However, little related work study star join in Column store and MapReduce envi-ronments. This paper proposes two new cache conscious algorithms Multi-Fragment-Replication Join (MFRJ) and MapReduce-Invisible Join (MRIJ) in MapReduce environ-ments. All these algorithms avoid fact table data movement and are cache conscious in each MapReduce node. In ad-dition, fact table is partitioned into several Column groups for cache optimization in MFRJ; One group contains all of Foreign Key Columns and each measure Column is a group. In MRIJ, each Column is separately processed one by one which has higher cache utilization and avoids frequently cache miss from one Column to the other Column. MRIJ is composed of several map operation on dimension tables and one MapRe-duce job. We also apply MRIJ on RCFile in Hive. All oper-ations are processed in mapping phase and avoid high cost of shuffle and reduce operation. If the dimension tables are big enough and cannot cache in local memory, MRIJ is di-vided into two phases, firstly each dimension table join with corresponding Foreign Key Column in fact table as commonly map reduce join concurrently or serially; secondly all inter-nal results joined for final results based on position index. This strategy also can be applied to other multi-table join. In order to reduce network I/O, dimension table and the fact table Foreign Key Column are co-location storage. Our experimental results in cluster environments show that our algorithms outperform existing approaches in Hive system. Categories and Subject Descriptor