The Experts below are selected from a list of 2520 Experts worldwide ranked by ideXlab platform

Francisco Herrera - One of the best experts on this subject based on the ideXlab platform.

  • DPASF: a flink library for streaming data preprocessing
    Big Data Analytics, 2019
    Co-Authors: Alejandro Alcalde-barros, Diego García-gil, Salvador García, Francisco Herrera
    Abstract:

    Background Data preprocessing techniques are devoted to correcting or alleviating errors in data. Discretization and feature selection are two of the most extended data preprocessing techniques. Although we can find many proposals for static Big Data preprocessing, there is little research devoted to the continuous Big Data problem. Apache Flink is a recent and novel Big Data framework, following the Mapreduce Paradigm, focused on distributed stream and batch data processing. In this paper, we propose a data stream library for Big Data preprocessing, named DPASF, under Apache Flink. The library is composed of six of the most popular and widely used data preprocessing algorithms. It contains three algorithms for discretization, and three algorithms for performing feature selection. Results The algorithms have been tested using two Big Data datasets. Experimental results show that preprocessing can not only reduce the size of the data, but also maintain or even improve the original accuracy in a short period of time. Conclusion DPASF contains algorithms that are useful when dealing with Big Data data streams. The preprocessing algorithms included in the library are able to tackle Big Datasets efficiently and to correct imperfections in the data.

  • evolutionary undersampling for extremely imbalanced big data classification under apache spark
    Congress on Evolutionary Computation, 2016
    Co-Authors: Isaac Triguero, Mikel Galar, D Merino, Jesus Maillo, Humberto Bustince, Francisco Herrera
    Abstract:

    The classification of datasets with a skewed class distribution is an important problem in data mining. Evolutionary undersampling of the majority class has proved to be a successful approach to tackle this issue. Such a challenging task may become even more difficult when the number of the majority class examples is very big. In this scenario, the use of the evolutionary model becomes unpractical due to the memory and time constrictions. Divide-and-conquer approaches based on the Mapreduce Paradigm have already been proposed to handle this type of problems by dividing data into multiple subsets. However, in extremely imbalanced cases, these models may suffer from a lack of density from the minority class in the subsets considered. Aiming at addressing this problem, in this contribution we provide a new big data scheme based on the new emerging technology Apache Spark to tackle highly imbalanced datasets. We take advantage of its in-memory operations to diminish the effect of the small sample size. The key point of this proposal lies in the independent management of majority and minority class examples, allowing us to keep a higher number of minority class examples in each subset. In our experiments, we analyze the proposed model with several data sets with up to 17 million instances. The results show the goodness of this evolutionary undersampling model for extremely imbalanced big data classification.

  • evolutionary feature selection for big data classification a Mapreduce approach
    Mathematical Problems in Engineering, 2015
    Co-Authors: Daniel Peralta, Sara Del Rio, Sergio Ramirezgallego, Isaac Triguero, Jose Manuel Benitez, Francisco Herrera
    Abstract:

    Nowadays, many disciplines have to deal with big datasets that additionally involve a high number of features. Feature selection methods aim at eliminating noisy, redundant, or irrelevant features that may deteriorate the classification performance. However, traditional methods lack enough scalability to cope with datasets of millions of instances and extract successful results in a delimited time. This paper presents a feature selection algorithm based on evolutionary computation that uses the Mapreduce Paradigm to obtain subsets of features from big datasets. The algorithm decomposes the original dataset in blocks of instances to learn from them in the map phase; then, the reduce phase merges the obtained partial results into a final vector of feature weights, which allows a flexible application of the feature selection procedure using a threshold to determine the selected subset of features. The feature selection method is evaluated by using three well-known classifiers (SVM, Logistic Regression, and Naive Bayes) implemented within the Spark framework to address big data problems. In the experiments, datasets up to 67 millions of instances and up to 2000 attributes have been managed, showing that this is a suitable framework to perform evolutionary feature selection, improving both the classification accuracy and its runtime when dealing with big data problems.

Jianping Fan - One of the best experts on this subject based on the ideXlab platform.

  • Mrdbscan: An efficient parallel density-based clustering algorithm using Mapreduce
    2020
    Co-Authors: Haoyu Tan, Wuman Luo, Huajian Mao, Shengzhong Feng, Jianping Fan
    Abstract:

    Abstract-Data clustering is an important data mining technology that plays a crucial role in numerous scientific applications. However, it is challenging due to the size of datasets has been growing rapidly to extra-large scale in the real world. Meanwhile, Mapreduce is a desirable parallel programming platform that is widely applied in kinds of data process fields. In this paper, we propose an efficient parallel density-based clustering algorithm and implement it by a 4-stages Mapreduce Paradigm. Furthermore, we adopt a quick partitioning strategy for large scale non-indexed data. We study the metric of merge among bordering partitions and make optimizations on it. At last, we evaluate our work on real large scale datasets using Hadoop platform. Results reveal that the speedup and scaleup of our work are very efficient

  • mr dbscan an efficient parallel density based clustering algorithm using Mapreduce
    International Conference on Parallel and Distributed Systems, 2011
    Co-Authors: Haoyu Tan, Wuman Luo, Huajian Mao, Shengzhong Feng, Jianping Fan
    Abstract:

    Data clustering is an important data mining technology that plays a crucial role in numerous scientific applications. However, it is challenging due to the size of datasets has been growing rapidly to extra-large scale in the real world. Meanwhile, Mapreduce is a desirable parallel programming platform that is widely applied in kinds of data process fields. In this paper, we propose an efficient parallel density-based clustering algorithm and implement it by a 4-stages Mapreduce Paradigm. Furthermore, we adopt a quick partitioning strategy for large scale non-indexed data. We study the metric of merge among bordering partitions and make optimizations on it. At last, we evaluate our work on real large scale datasets using Hadoop platform. Results reveal that the speedup and scale up of our work are very efficient.

Isaac Triguero - One of the best experts on this subject based on the ideXlab platform.

  • evolutionary undersampling for extremely imbalanced big data classification under apache spark
    Congress on Evolutionary Computation, 2016
    Co-Authors: Isaac Triguero, Mikel Galar, D Merino, Jesus Maillo, Humberto Bustince, Francisco Herrera
    Abstract:

    The classification of datasets with a skewed class distribution is an important problem in data mining. Evolutionary undersampling of the majority class has proved to be a successful approach to tackle this issue. Such a challenging task may become even more difficult when the number of the majority class examples is very big. In this scenario, the use of the evolutionary model becomes unpractical due to the memory and time constrictions. Divide-and-conquer approaches based on the Mapreduce Paradigm have already been proposed to handle this type of problems by dividing data into multiple subsets. However, in extremely imbalanced cases, these models may suffer from a lack of density from the minority class in the subsets considered. Aiming at addressing this problem, in this contribution we provide a new big data scheme based on the new emerging technology Apache Spark to tackle highly imbalanced datasets. We take advantage of its in-memory operations to diminish the effect of the small sample size. The key point of this proposal lies in the independent management of majority and minority class examples, allowing us to keep a higher number of minority class examples in each subset. In our experiments, we analyze the proposed model with several data sets with up to 17 million instances. The results show the goodness of this evolutionary undersampling model for extremely imbalanced big data classification.

  • evolutionary feature selection for big data classification a Mapreduce approach
    Mathematical Problems in Engineering, 2015
    Co-Authors: Daniel Peralta, Sara Del Rio, Sergio Ramirezgallego, Isaac Triguero, Jose Manuel Benitez, Francisco Herrera
    Abstract:

    Nowadays, many disciplines have to deal with big datasets that additionally involve a high number of features. Feature selection methods aim at eliminating noisy, redundant, or irrelevant features that may deteriorate the classification performance. However, traditional methods lack enough scalability to cope with datasets of millions of instances and extract successful results in a delimited time. This paper presents a feature selection algorithm based on evolutionary computation that uses the Mapreduce Paradigm to obtain subsets of features from big datasets. The algorithm decomposes the original dataset in blocks of instances to learn from them in the map phase; then, the reduce phase merges the obtained partial results into a final vector of feature weights, which allows a flexible application of the feature selection procedure using a threshold to determine the selected subset of features. The feature selection method is evaluated by using three well-known classifiers (SVM, Logistic Regression, and Naive Bayes) implemented within the Spark framework to address big data problems. In the experiments, datasets up to 67 millions of instances and up to 2000 attributes have been managed, showing that this is a suitable framework to perform evolutionary feature selection, improving both the classification accuracy and its runtime when dealing with big data problems.

David Camacho - One of the best experts on this subject based on the ideXlab platform.

  • social big data
    Information Fusion, 2016
    Co-Authors: Gema Belloorgaz, Jason J Jung, David Camacho
    Abstract:

    The paper presents the methodologies on information fusion for social media.The methodologies, frameworks, and software used to work with big data are given.The state of the art in the data analytic techniques on social big data is provided.Social big data applications for various domains are described and analyzed. Big data has become an important issue for a large number of research areas such as data mining, machine learning, computational intelligence, information fusion, the semantic Web, and social networks. The rise of different big data frameworks such as Apache Hadoop and, more recently, Spark, for massive data processing based on the Mapreduce Paradigm has allowed for the efficient utilisation of data mining methods and machine learning algorithms in different domains. A number of libraries such as Mahout and SparkMLib have been designed to develop new efficient applications based on machine learning algorithms. The combination of big data technologies and traditional machine learning algorithms has generated new and interesting challenges in other areas as social media and social networks. These new challenges are focused mainly on problems such as data processing, data storage, data representation, and how data can be used for pattern mining, analysing user behaviours, and visualizing and tracking data, among others. In this paper, we present a revision of the new methodologies that is designed to allow for efficient data mining and information fusion from social media and of the new applications and frameworks that are currently appearing under the "umbrella" of the social networks, social media and big data Paradigms.

Haoyu Tan - One of the best experts on this subject based on the ideXlab platform.

  • Mrdbscan: An efficient parallel density-based clustering algorithm using Mapreduce
    2020
    Co-Authors: Haoyu Tan, Wuman Luo, Huajian Mao, Shengzhong Feng, Jianping Fan
    Abstract:

    Abstract-Data clustering is an important data mining technology that plays a crucial role in numerous scientific applications. However, it is challenging due to the size of datasets has been growing rapidly to extra-large scale in the real world. Meanwhile, Mapreduce is a desirable parallel programming platform that is widely applied in kinds of data process fields. In this paper, we propose an efficient parallel density-based clustering algorithm and implement it by a 4-stages Mapreduce Paradigm. Furthermore, we adopt a quick partitioning strategy for large scale non-indexed data. We study the metric of merge among bordering partitions and make optimizations on it. At last, we evaluate our work on real large scale datasets using Hadoop platform. Results reveal that the speedup and scaleup of our work are very efficient

  • mr dbscan an efficient parallel density based clustering algorithm using Mapreduce
    International Conference on Parallel and Distributed Systems, 2011
    Co-Authors: Haoyu Tan, Wuman Luo, Huajian Mao, Shengzhong Feng, Jianping Fan
    Abstract:

    Data clustering is an important data mining technology that plays a crucial role in numerous scientific applications. However, it is challenging due to the size of datasets has been growing rapidly to extra-large scale in the real world. Meanwhile, Mapreduce is a desirable parallel programming platform that is widely applied in kinds of data process fields. In this paper, we propose an efficient parallel density-based clustering algorithm and implement it by a 4-stages Mapreduce Paradigm. Furthermore, we adopt a quick partitioning strategy for large scale non-indexed data. We study the metric of merge among bordering partitions and make optimizations on it. At last, we evaluate our work on real large scale datasets using Hadoop platform. Results reveal that the speedup and scale up of our work are very efficient.