The Experts below are selected from a list of 21975 Experts worldwide ranked by ideXlab platform
Lei Shi - One of the best experts on this subject based on the ideXlab platform.
-
a tile based scalable Raster Data management system based on hdfs
International Conference on Geoinformatics, 2012Co-Authors: Guangqing Zhang, Chuanjie Xie, Lei ShiAbstract:Hadoop has become a worldwide popular open source platform for large Data analysis in commercial application and Hadoop distributed file system (HDFS) is the core part of it. However, HDFS cannot be used directly for managing Raster Data, for the geographic location information is involved. In this paper, we describe the implementation of a tile-based scalable Raster Data management system based on HDFS. While reserving the basic architecture of HDFS, we reorganize the Data structure in block, add some additional metaData, design an index Data structure in block, keep an overlapping region between adjacent blocks, and offer a compression option for users. Besides, we provide functions for reading the Raster Data from HDFS in tile stream. These optimizations match the feature of Raster Data to the architecture of HDFS. MapReduce Applications can be built on the Raster Data management system.
-
Geoinformatics - A tile-based scalable Raster Data management system based on HDFS
2012 20th International Conference on Geoinformatics, 2012Co-Authors: Guangqing Zhang, Chuanjie Xie, Lei ShiAbstract:Hadoop has become a worldwide popular open source platform for large Data analysis in commercial application and Hadoop distributed file system (HDFS) is the core part of it. However, HDFS cannot be used directly for managing Raster Data, for the geographic location information is involved. In this paper, we describe the implementation of a tile-based scalable Raster Data management system based on HDFS. While reserving the basic architecture of HDFS, we reorganize the Data structure in block, add some additional metaData, design an index Data structure in block, keep an overlapping region between adjacent blocks, and offer a compression option for users. Besides, we provide functions for reading the Raster Data from HDFS in tile stream. These optimizations match the feature of Raster Data to the architecture of HDFS. MapReduce Applications can be built on the Raster Data management system.
Alexis J. Comber - One of the best experts on this subject based on the ideXlab platform.
-
Investigating spatial error structures in continuous Raster Data
International Journal of Applied Earth Observation and Geoinformation, 2019Co-Authors: Narumasa Tsutsumida, Pedro Rodriguez-veiga, Paul Harris, Heiko Balzter, Alexis J. ComberAbstract:The objective of this study is to investigate spatial structures of error in the assessment of continuous Raster Data. The use of conventional diagnostics of error often overlooks the possible spatial variation in error because such diagnostics report only average error or deviation between predicted and reference values. In this respect, this work uses a moving window (kernel) approach to generate geographically weighted (GW) versions of the mean signed deviation, the mean absolute error and the root mean squared error and to quantify their spatial variations. Such approach computes local error diagnostics from Data weighted by its distance to the centre of a moving kernel and allows to map spatial surfaces of each type of error. In addition, a GW correlation analysis between predicted and reference values provides an alternative view of local error. These diagnostics are applied to two earth observation case studies. The results reveal important spatial structures of error and unusual clusters of error can be identified through Monte Carlo permutation tests. The first case study demonstrates the use of GW diagnostics to fractional impervious surface area Datasets generated by four different models for the Jakarta metropolitan area, Indonesia. The GW diagnostics reveal where the models perform differently and similarly, and found areas of under-prediction in the urban core, with larger errors in peri-urban areas. The second case study uses the GW diagnostics to four remotely sensed aboveground biomass Datasets for the Yucatan Peninsula, Mexico. The mapping of GW diagnostics provides a means to compare the accuracy of these four continuous Raster Datasets locally. The discussion considers the relative nature of diagnostics of error, determining moving window size and issues around the interpretation of different error diagnostic measures. Investigating spatial structures of error hidden in conventional diagnostics of error provides informative descriptions of error in continuous Raster Data.
-
Investigating Spatial Error Structures in Continuous Raster Data
arXiv: Applications, 2018Co-Authors: Narumasa Tsutsumida, Pedro Rodriguez-veiga, Paul Harris, Heiko Balzter, Alexis J. ComberAbstract:The objective of this study is to investigate spatial structures of error in the assessment of continuous Raster Data. The use of conventional diagnostics of error often overlooks the possible spatial variation in error because such diagnostics report only average error or deviation between predicted and reference values. In this respect, this work uses a moving window (kernel) approach to generate geographically weighted (GW) versions of the mean signed deviation, the mean absolute error and the root mean squared error and to quantify their spatial variations. Such approach computes local error diagnostics from Data weighted by its distance to the centre of a moving kernel and allows to map spatial surfaces of each type of error. In addition, a GW correlation analysis between predicted and reference values provides an alternative view of local error. Full abstract can be found in the pdf.
Junzhen Meng - One of the best experts on this subject based on the ideXlab platform.
-
Raster Data projection transformation based-on Kriging interpolation approximate grid algorithm
Alexandria Engineering Journal, 1Co-Authors: Junzhen MengAbstract:Abstract To solve the problems of small area, slow calculation speed and low precision in the projection transformation algorithm of Raster Data, the idea of using Kriging interpolation approximate grid algorithm for Raster Data projection transformation is proposed. Through the experimental results of point-to-point transformation and Kriging interpolation approximate grid algorithm transformation under the same conditions, it can be seen that for different projection types under the same limit conditions, Kriging interpolation approximate grid algorithm can ensure that the Raster Data projection error is always within the given projection limit. With the same number of points, the Kriging interpolation approximate grid algorithm is faster and more efficient than the point-to-point projection algorithm. Under the same pixel condition, the Kriging interpolation approximate grid algorithm is faster, more accurate and more effective than the point-to-point projection algorithm.
Ramon Antonio Rodriges Zalipynis - One of the best experts on this subject based on the ideXlab platform.
-
Ershov Informatics Conference - Distributed In Situ Processing of Big Raster Data in the Cloud
Lecture Notes in Computer Science, 2018Co-Authors: Ramon Antonio Rodriges ZalipynisAbstract:A Raster is the primary Data type in Earth science, geology, remote sensing and other fields with tremendous growth of Data volumes. An array DBMS is an option to tackle big Raster Data processing. However, Raster Data are traditionally stored in files, not in Databases. Command line tools have long being developed to process Raster files. Most tools are feature-rich and free but optimized for a single machine. This paper proposes new techniques for distributed processing of Raster Data directly in diverse file formats by delegating considerable portions of work to such tools. An N-dimensional array Data model is proposed to maintain independence from the files and the tools. Also, a new scheme named GROUP–APPLY–FINALLY is presented to universally express the majority of Raster Data processing operations and streamline their distributed execution. New approaches make it possible to provide a rich collection of Raster operations at scale and outperform SciDB over \(410\times \) on average on climate reanalysis Data. SciDB is the only freely available distributed array DBMS to date. Experiments were carried out on 8- and 16-node clusters in Microsoft Azure Cloud.
-
AIST - Array DBMS and Satellite Imagery: Towards Big Raster Data in the Cloud
Lecture Notes in Computer Science, 2017Co-Authors: Ramon Antonio Rodriges Zalipynis, Evgeniy Pozdeev, Anton BryukhovAbstract:Satellite imagery have always been “big” Data. Array DBMS is one of the tools to streamline Raster Data processing. However, Raster Data are usually stored in files, not in Databases. Respective command line tools have long been developed to process these files. Most of the tools are feature-rich and free but optimized for a single machine. The approach of partially delegating in situ Raster Data processing to such tools has been recently proposed. The approach includes a new formal N-d array Data model to abstract from the files and the tools as well as new formal distributed algorithms based on the model. ChronosServer is a distributed array DBMS under development into which the approach is being integrated. This paper extends the approach with a new algorithm for the reshaping (tiling) of arbitrary N-d arrays onto a set of overlapping N-d arrays with a fixed shape. Cutting arrays with an overlap enables to perform a broad range of large imagery processing operations in a distributed shared-nothing fashion. Currently ChronosServer provides a rich collection of Raster operations at scale and outperforms SciDB up to 80\(\times \) on Landsat Data. SciDB is the only freely available distributed array DBMS to date. Experiments were carried out on 8- and 16-node clusters in Microsoft Azure Cloud.
-
distributed in situ processing of big Raster Data in the cloud
International Andrei Ershov Memorial Conference on Perspectives of System Informatics, 2017Co-Authors: Ramon Antonio Rodriges ZalipynisAbstract:A Raster is the primary Data type in Earth science, geology, remote sensing and other fields with tremendous growth of Data volumes. An array DBMS is an option to tackle big Raster Data processing. However, Raster Data are traditionally stored in files, not in Databases. Command line tools have long being developed to process Raster files. Most tools are feature-rich and free but optimized for a single machine. This paper proposes new techniques for distributed processing of Raster Data directly in diverse file formats by delegating considerable portions of work to such tools. An N-dimensional array Data model is proposed to maintain independence from the files and the tools. Also, a new scheme named GROUP–APPLY–FINALLY is presented to universally express the majority of Raster Data processing operations and streamline their distributed execution. New approaches make it possible to provide a rich collection of Raster operations at scale and outperform SciDB over \(410\times \) on average on climate reanalysis Data. SciDB is the only freely available distributed array DBMS to date. Experiments were carried out on 8- and 16-node clusters in Microsoft Azure Cloud.
Guangqing Zhang - One of the best experts on this subject based on the ideXlab platform.
-
a tile based scalable Raster Data management system based on hdfs
International Conference on Geoinformatics, 2012Co-Authors: Guangqing Zhang, Chuanjie Xie, Lei ShiAbstract:Hadoop has become a worldwide popular open source platform for large Data analysis in commercial application and Hadoop distributed file system (HDFS) is the core part of it. However, HDFS cannot be used directly for managing Raster Data, for the geographic location information is involved. In this paper, we describe the implementation of a tile-based scalable Raster Data management system based on HDFS. While reserving the basic architecture of HDFS, we reorganize the Data structure in block, add some additional metaData, design an index Data structure in block, keep an overlapping region between adjacent blocks, and offer a compression option for users. Besides, we provide functions for reading the Raster Data from HDFS in tile stream. These optimizations match the feature of Raster Data to the architecture of HDFS. MapReduce Applications can be built on the Raster Data management system.
-
Geoinformatics - A tile-based scalable Raster Data management system based on HDFS
2012 20th International Conference on Geoinformatics, 2012Co-Authors: Guangqing Zhang, Chuanjie Xie, Lei ShiAbstract:Hadoop has become a worldwide popular open source platform for large Data analysis in commercial application and Hadoop distributed file system (HDFS) is the core part of it. However, HDFS cannot be used directly for managing Raster Data, for the geographic location information is involved. In this paper, we describe the implementation of a tile-based scalable Raster Data management system based on HDFS. While reserving the basic architecture of HDFS, we reorganize the Data structure in block, add some additional metaData, design an index Data structure in block, keep an overlapping region between adjacent blocks, and offer a compression option for users. Besides, we provide functions for reading the Raster Data from HDFS in tile stream. These optimizations match the feature of Raster Data to the architecture of HDFS. MapReduce Applications can be built on the Raster Data management system.
