The Experts below are selected from a list of 928683 Experts worldwide ranked by ideXlab platform
Xingli Yang - One of the best experts on this subject based on the ideXlab platform.
-
confidence interval for bm f_1 measure of Algorithm Performance based on blocked 3 bm times 2 cross validation
IEEE Transactions on Knowledge and Data Engineering, 2015Co-Authors: Yu Wang, Jihong Li, Yanfang Li, Ruibo Wang, Xingli YangAbstract:In studies on the application of machine learning such as Information Retrieval (IR), the focus is typically on the estimation of the $F_1$ measure of Algorithm Performance. Approximate symmetrical confidence intervals constructed by the $F_1$ value based on cross-validated $t$ distribution are commonly used in the literature. However, theoretical analysis on the distribution of $F_1$ values shows that such distribution is actually non-symmetrical. Thus, simply using symmetrical distribution to approximate non-symmetrical distribution may be inappropriate and may result in a low degree of confidence and long interval length for the confidence interval. In the present study, a non-symmetrical confidence interval of the $F_1$ measure based on Beta prime distribution is constructed by using the $F_1$ value computed based on the average confusion matrix of a blocked $3\times2$ cross-validation. Experimental results show that in most cases, our method has high degrees of confidence. With an acceptable degree of confidence, our method has a shorter interval length than the approximate symmetrical confidence intervals based on the blocked $3\times 2$ and $5 \times 2$ cross-validated $t$ distributions. The approximate symmetrical confidence interval based on the $10$ -fold cross-validated $t$ distribution has the shortest interval length of the four confidence intervals but with low degrees of confidence in all cases. Taking these two factors into consideration, our method is recommended.
-
Confidence Interval for ${\bm {F_1}}$ Measure of Algorithm Performance Based on Blocked 3 $\bm {\times}$ 2 Cross-Validation
IEEE Transactions on Knowledge and Data Engineering, 2015Co-Authors: Yu Wang, Ruibo Wang, Xingli YangAbstract:In studies on the application of machine learning such as Information Retrieval (IR), the focus is typically on the estimation of the $F_1$ measure of Algorithm Performance. Approximate symmetrical confidence intervals constructed by the $F_1$ value based on cross-validated $t$ distribution are commonly used in the literature. However, theoretical analysis on the distribution of $F_1$ values shows that such distribution is actually non-symmetrical. Thus, simply using symmetrical distribution to approximate non-symmetrical distribution may be inappropriate and may result in a low degree of confidence and long interval length for the confidence interval. In the present study, a non-symmetrical confidence interval of the $F_1$ measure based on Beta prime distribution is constructed by using the $F_1$ value computed based on the average confusion matrix of a blocked $3\times2$ cross-validation. Experimental results show that in most cases, our method has high degrees of confidence. With an acceptable degree of confidence, our method has a shorter interval length than the approximate symmetrical confidence intervals based on the blocked $3\times 2$ and $5 \times 2$ cross-validated $t$ distributions. The approximate symmetrical confidence interval based on the $10$ -fold cross-validated $t$ distribution has the shortest interval length of the four confidence intervals but with low degrees of confidence in all cases. Taking these two factors into consideration, our method is recommended.
Yu Wang - One of the best experts on this subject based on the ideXlab platform.
-
confidence interval for bm f_1 measure of Algorithm Performance based on blocked 3 bm times 2 cross validation
IEEE Transactions on Knowledge and Data Engineering, 2015Co-Authors: Yu Wang, Jihong Li, Yanfang Li, Ruibo Wang, Xingli YangAbstract:In studies on the application of machine learning such as Information Retrieval (IR), the focus is typically on the estimation of the $F_1$ measure of Algorithm Performance. Approximate symmetrical confidence intervals constructed by the $F_1$ value based on cross-validated $t$ distribution are commonly used in the literature. However, theoretical analysis on the distribution of $F_1$ values shows that such distribution is actually non-symmetrical. Thus, simply using symmetrical distribution to approximate non-symmetrical distribution may be inappropriate and may result in a low degree of confidence and long interval length for the confidence interval. In the present study, a non-symmetrical confidence interval of the $F_1$ measure based on Beta prime distribution is constructed by using the $F_1$ value computed based on the average confusion matrix of a blocked $3\times2$ cross-validation. Experimental results show that in most cases, our method has high degrees of confidence. With an acceptable degree of confidence, our method has a shorter interval length than the approximate symmetrical confidence intervals based on the blocked $3\times 2$ and $5 \times 2$ cross-validated $t$ distributions. The approximate symmetrical confidence interval based on the $10$ -fold cross-validated $t$ distribution has the shortest interval length of the four confidence intervals but with low degrees of confidence in all cases. Taking these two factors into consideration, our method is recommended.
-
Confidence Interval for ${\bm {F_1}}$ Measure of Algorithm Performance Based on Blocked 3 $\bm {\times}$ 2 Cross-Validation
IEEE Transactions on Knowledge and Data Engineering, 2015Co-Authors: Yu Wang, Ruibo Wang, Xingli YangAbstract:In studies on the application of machine learning such as Information Retrieval (IR), the focus is typically on the estimation of the $F_1$ measure of Algorithm Performance. Approximate symmetrical confidence intervals constructed by the $F_1$ value based on cross-validated $t$ distribution are commonly used in the literature. However, theoretical analysis on the distribution of $F_1$ values shows that such distribution is actually non-symmetrical. Thus, simply using symmetrical distribution to approximate non-symmetrical distribution may be inappropriate and may result in a low degree of confidence and long interval length for the confidence interval. In the present study, a non-symmetrical confidence interval of the $F_1$ measure based on Beta prime distribution is constructed by using the $F_1$ value computed based on the average confusion matrix of a blocked $3\times2$ cross-validation. Experimental results show that in most cases, our method has high degrees of confidence. With an acceptable degree of confidence, our method has a shorter interval length than the approximate symmetrical confidence intervals based on the blocked $3\times 2$ and $5 \times 2$ cross-validated $t$ distributions. The approximate symmetrical confidence interval based on the $10$ -fold cross-validated $t$ distribution has the shortest interval length of the four confidence intervals but with low degrees of confidence in all cases. Taking these two factors into consideration, our method is recommended.
Laura Barbulescu - One of the best experts on this subject based on the ideXlab platform.
-
Understanding Algorithm Performance on an oversubscribed scheduling application
Journal of Artificial Intelligence Research, 2006Co-Authors: Laura Barbulescu, Adele E. Howe, L. Darrell Whitley, Mark RobertsAbstract:The best performing Algorithms for a particular oversubscribed scheduling application, Air Force Satellite Control Network (AFSCN) scheduling, appear to have little in common. Yet, through careful experimentation and modeling of Performance in real problem instances, we can relate characteristics of the best Algorithms to characteristics of the application. In particular, we find that plateaus dominate the search spaces (thus favoring Algorithms that make larger changes to solutions) and that some randomization in exploration is critical to good Performance (due to the lack of gradient information on the plateaus). Based on our explanations of Algorithm Performance, we develop a new Algorithm that combines characteristics of the best performers; the new Algorithm's Performance is better than the previous best. We show how hypothesis driven experimentation and search modeling can both explain Algorithm Performance and motivate the design of a new Algorithm.
-
Analyses of Algorithm Performance for an oversubscribed scheduling problem
2005Co-Authors: Darrell Whitley, Adele E. Howe, Laura BarbulescuAbstract:We analyze the factors that influence Algorithm Performance for an oversubscribed application, scheduling for the Air Force Satellite Control Network (AFSCN). AFSCN scheduling assigns access requests to specific time slots on antennas at ground stations. The application is oversubscribed: not all tasks can be accommodated given the available resources. As a special class of scheduling problems, oversubscribed problems present additional challenges. While, in general, solutions to scheduling problems specify the start times and resources assigned to tasks, in oversubscribed scheduling the maximal subset of tasks that can be scheduled with the available resources also needs to be identified. We implemented various Algorithms for AFSCN scheduling. Some Algorithms, such as a domain-specific repair-based Algorithm or constraint-based scheduling heuristics, failed to identify good solutions. We have found a set of fairly simple Algorithms that perform well on the AFSCN scheduling domain, for both real and synthetically generated problems. The Algorithms in the set are: hill-climbing, a genetic Algorithm (GA) and Squeaky Wheel Optimization (SWO). All the Algorithms in the set are designed to traverse the same search space: solutions are represented as permutations of tasks; a greedy schedule builder converts the permutation into a schedule by assigning a start time and resources to the requests in the order in which they appear in the permutation. However, these Algorithms vary in the way they traverse the search space. This research identifies Performance factors that make each of the Algorithms a good fit for AFSCN scheduling. The AFSCN scheduling search space is dominated by plateaus, due to both the discrete nature of the objective function and to the fact that the schedule builder converts multiple permutations into identical schedules. Each Algorithm handles plateaus differently. Hill-climbing randomly walks on the plateaus until it finds exits to lower plateaus: the higher the percentage of the space occupied by plateaus, the more random wandering is likely for hill-climbing. We found the ordering of the neighbors to be the main Performance factor in expediting plateau traversal for hill-climbing. The GA and SWO both traverse the plateaus quickly, by making multiple changes to the solutions. The long, directed leaps across the search space are the main Performance factor for the GA and SWO. We also investigated whether initializing the search closer to the best solutions is the key to Performance. We found that such initialization helps but is not by itself enough to explain Algorithm Performance results. The main contributions of this research work are: (1) We performed the first coupled formal and empirical analysis of the AFSCN scheduling problem. (2) We designed techniques for analyzing Algorithm Performance, which could transfer to other applications. (3) We identified Algorithm Performance factors, which are likely to hold on other similar problems. (4) We designed a new best performing Algorithm, by combining the features we found to have most influence on Performance.
-
contrasting structured and random permutation flow shop scheduling problems search space topology and Algorithm Performance
Informs Journal on Computing, 2002Co-Authors: Jean-paul Watson, Laura Barbulescu, Darrell L Whitley, Adele E. HoweAbstract:The use of random test problems to evaluate Algorithm Performance raises an important, and generally unanswered, question: Are the results generalizable to more realistic problems? Researchers generally assume that Algorithms with superior Performance on difficult, random test problems will also perform well on more realistic, structured problems. Our research explores this assumption for the permutation flow-shop scheduling problem. We introduce a method for generating structured flow-shop problems, which are modeled after features found in some real-world manufacturing environments. We perform experiments that indicate significant differences exist between the search-space topologies of random and structured flow-shop problems, and demonstrate that these differencescan affect the Performance of certain Algorithms. Yet despite these differences, and in contrast to difficult random problems, the majority of structured flow-shop problems were easily solved to optimality by most Algorithms. For the problems not optimally solved, differences in Performance were minor. We conclude that more realistic, structured permutation flow-shop problems are actually relatively easy to solve. Our results also raise doubts as to whether superior Performance on difficult random scheduling problems translates into superior Performance on more realistic kinds of scheduling problems.
-
AAAI/IAAI - Algorithm Performance and problem structure for flow-shop scheduling
1999Co-Authors: Jean-paul Watson, Laura Barbulescu, Adele E. Howe, L. Darrell WhitleyAbstract:Test suites for many domains often fail to model features present in real-world problems. For the permutation flow-shop sequencing problem (PFSP), the most popular test suite consists of problems whose features are generated from a single uniform random distribution. Synthetic generation of problems with characteristics present in real-world problems is a viable alternative. We compare the Performance of several competitive Algorithms on problems produced with such a generator. We find that, as more realistic characteristics are introduced, the Performance of a state-of-the-art Algorithm degrades rapidly: faster and less complex stochastic Algorithms provide superior Performance. Our empirical results show that small changes in problem structure or problem size can influence Algorithm Performance. We hypothesize that these Performance differences may be partially due to differences in search space topologies; we show that structured problems produce topologies with Performance plateaus. Algorithm sensitivity to problem charaeteristics suggests the need to construct test suites more representative of real-world applications.
Adele E. Howe - One of the best experts on this subject based on the ideXlab platform.
-
Understanding Algorithm Performance on an oversubscribed scheduling application
Journal of Artificial Intelligence Research, 2006Co-Authors: Laura Barbulescu, Adele E. Howe, L. Darrell Whitley, Mark RobertsAbstract:The best performing Algorithms for a particular oversubscribed scheduling application, Air Force Satellite Control Network (AFSCN) scheduling, appear to have little in common. Yet, through careful experimentation and modeling of Performance in real problem instances, we can relate characteristics of the best Algorithms to characteristics of the application. In particular, we find that plateaus dominate the search spaces (thus favoring Algorithms that make larger changes to solutions) and that some randomization in exploration is critical to good Performance (due to the lack of gradient information on the plateaus). Based on our explanations of Algorithm Performance, we develop a new Algorithm that combines characteristics of the best performers; the new Algorithm's Performance is better than the previous best. We show how hypothesis driven experimentation and search modeling can both explain Algorithm Performance and motivate the design of a new Algorithm.
-
Analyses of Algorithm Performance for an oversubscribed scheduling problem
2005Co-Authors: Darrell Whitley, Adele E. Howe, Laura BarbulescuAbstract:We analyze the factors that influence Algorithm Performance for an oversubscribed application, scheduling for the Air Force Satellite Control Network (AFSCN). AFSCN scheduling assigns access requests to specific time slots on antennas at ground stations. The application is oversubscribed: not all tasks can be accommodated given the available resources. As a special class of scheduling problems, oversubscribed problems present additional challenges. While, in general, solutions to scheduling problems specify the start times and resources assigned to tasks, in oversubscribed scheduling the maximal subset of tasks that can be scheduled with the available resources also needs to be identified. We implemented various Algorithms for AFSCN scheduling. Some Algorithms, such as a domain-specific repair-based Algorithm or constraint-based scheduling heuristics, failed to identify good solutions. We have found a set of fairly simple Algorithms that perform well on the AFSCN scheduling domain, for both real and synthetically generated problems. The Algorithms in the set are: hill-climbing, a genetic Algorithm (GA) and Squeaky Wheel Optimization (SWO). All the Algorithms in the set are designed to traverse the same search space: solutions are represented as permutations of tasks; a greedy schedule builder converts the permutation into a schedule by assigning a start time and resources to the requests in the order in which they appear in the permutation. However, these Algorithms vary in the way they traverse the search space. This research identifies Performance factors that make each of the Algorithms a good fit for AFSCN scheduling. The AFSCN scheduling search space is dominated by plateaus, due to both the discrete nature of the objective function and to the fact that the schedule builder converts multiple permutations into identical schedules. Each Algorithm handles plateaus differently. Hill-climbing randomly walks on the plateaus until it finds exits to lower plateaus: the higher the percentage of the space occupied by plateaus, the more random wandering is likely for hill-climbing. We found the ordering of the neighbors to be the main Performance factor in expediting plateau traversal for hill-climbing. The GA and SWO both traverse the plateaus quickly, by making multiple changes to the solutions. The long, directed leaps across the search space are the main Performance factor for the GA and SWO. We also investigated whether initializing the search closer to the best solutions is the key to Performance. We found that such initialization helps but is not by itself enough to explain Algorithm Performance results. The main contributions of this research work are: (1) We performed the first coupled formal and empirical analysis of the AFSCN scheduling problem. (2) We designed techniques for analyzing Algorithm Performance, which could transfer to other applications. (3) We identified Algorithm Performance factors, which are likely to hold on other similar problems. (4) We designed a new best performing Algorithm, by combining the features we found to have most influence on Performance.
-
contrasting structured and random permutation flow shop scheduling problems search space topology and Algorithm Performance
Informs Journal on Computing, 2002Co-Authors: Jean-paul Watson, Laura Barbulescu, Darrell L Whitley, Adele E. HoweAbstract:The use of random test problems to evaluate Algorithm Performance raises an important, and generally unanswered, question: Are the results generalizable to more realistic problems? Researchers generally assume that Algorithms with superior Performance on difficult, random test problems will also perform well on more realistic, structured problems. Our research explores this assumption for the permutation flow-shop scheduling problem. We introduce a method for generating structured flow-shop problems, which are modeled after features found in some real-world manufacturing environments. We perform experiments that indicate significant differences exist between the search-space topologies of random and structured flow-shop problems, and demonstrate that these differencescan affect the Performance of certain Algorithms. Yet despite these differences, and in contrast to difficult random problems, the majority of structured flow-shop problems were easily solved to optimality by most Algorithms. For the problems not optimally solved, differences in Performance were minor. We conclude that more realistic, structured permutation flow-shop problems are actually relatively easy to solve. Our results also raise doubts as to whether superior Performance on difficult random scheduling problems translates into superior Performance on more realistic kinds of scheduling problems.
-
AAAI/IAAI - Algorithm Performance and problem structure for flow-shop scheduling
1999Co-Authors: Jean-paul Watson, Laura Barbulescu, Adele E. Howe, L. Darrell WhitleyAbstract:Test suites for many domains often fail to model features present in real-world problems. For the permutation flow-shop sequencing problem (PFSP), the most popular test suite consists of problems whose features are generated from a single uniform random distribution. Synthetic generation of problems with characteristics present in real-world problems is a viable alternative. We compare the Performance of several competitive Algorithms on problems produced with such a generator. We find that, as more realistic characteristics are introduced, the Performance of a state-of-the-art Algorithm degrades rapidly: faster and less complex stochastic Algorithms provide superior Performance. Our empirical results show that small changes in problem structure or problem size can influence Algorithm Performance. We hypothesize that these Performance differences may be partially due to differences in search space topologies; we show that structured problems produce topologies with Performance plateaus. Algorithm sensitivity to problem charaeteristics suggests the need to construct test suites more representative of real-world applications.
Ruibo Wang - One of the best experts on this subject based on the ideXlab platform.
-
confidence interval for bm f_1 measure of Algorithm Performance based on blocked 3 bm times 2 cross validation
IEEE Transactions on Knowledge and Data Engineering, 2015Co-Authors: Yu Wang, Jihong Li, Yanfang Li, Ruibo Wang, Xingli YangAbstract:In studies on the application of machine learning such as Information Retrieval (IR), the focus is typically on the estimation of the $F_1$ measure of Algorithm Performance. Approximate symmetrical confidence intervals constructed by the $F_1$ value based on cross-validated $t$ distribution are commonly used in the literature. However, theoretical analysis on the distribution of $F_1$ values shows that such distribution is actually non-symmetrical. Thus, simply using symmetrical distribution to approximate non-symmetrical distribution may be inappropriate and may result in a low degree of confidence and long interval length for the confidence interval. In the present study, a non-symmetrical confidence interval of the $F_1$ measure based on Beta prime distribution is constructed by using the $F_1$ value computed based on the average confusion matrix of a blocked $3\times2$ cross-validation. Experimental results show that in most cases, our method has high degrees of confidence. With an acceptable degree of confidence, our method has a shorter interval length than the approximate symmetrical confidence intervals based on the blocked $3\times 2$ and $5 \times 2$ cross-validated $t$ distributions. The approximate symmetrical confidence interval based on the $10$ -fold cross-validated $t$ distribution has the shortest interval length of the four confidence intervals but with low degrees of confidence in all cases. Taking these two factors into consideration, our method is recommended.
-
Confidence Interval for ${\bm {F_1}}$ Measure of Algorithm Performance Based on Blocked 3 $\bm {\times}$ 2 Cross-Validation
IEEE Transactions on Knowledge and Data Engineering, 2015Co-Authors: Yu Wang, Ruibo Wang, Xingli YangAbstract:In studies on the application of machine learning such as Information Retrieval (IR), the focus is typically on the estimation of the $F_1$ measure of Algorithm Performance. Approximate symmetrical confidence intervals constructed by the $F_1$ value based on cross-validated $t$ distribution are commonly used in the literature. However, theoretical analysis on the distribution of $F_1$ values shows that such distribution is actually non-symmetrical. Thus, simply using symmetrical distribution to approximate non-symmetrical distribution may be inappropriate and may result in a low degree of confidence and long interval length for the confidence interval. In the present study, a non-symmetrical confidence interval of the $F_1$ measure based on Beta prime distribution is constructed by using the $F_1$ value computed based on the average confusion matrix of a blocked $3\times2$ cross-validation. Experimental results show that in most cases, our method has high degrees of confidence. With an acceptable degree of confidence, our method has a shorter interval length than the approximate symmetrical confidence intervals based on the blocked $3\times 2$ and $5 \times 2$ cross-validated $t$ distributions. The approximate symmetrical confidence interval based on the $10$ -fold cross-validated $t$ distribution has the shortest interval length of the four confidence intervals but with low degrees of confidence in all cases. Taking these two factors into consideration, our method is recommended.
