The Experts below are selected from a list of 12093 Experts worldwide ranked by ideXlab platform

Tim Menzies - One of the best experts on this subject based on the ideXlab platform.

  • Simpler Hyperparameter Optimization for Software Analytics: Why, How, When?
    arXiv: Software Engineering, 2020
    Co-Authors: Amritanshu Agrawal, Xueqi Yang, Rishabh Agrawal, Xipeng Shen, Tim Menzies
    Abstract:

    How to make Software Analytics simpler and faster? One method is to match the complexity of analysis to the intrinsic complexity of the data being explored. For example, hyperparameter optimizers find the control settings for data miners that improve for improving the predictions generated via Software Analytics. Sometimes, very fast hyperparameter optimization can be achieved by just DODGE-ing away from things tried before. But when is it wise to use DODGE and when must we use more complex (and much slower) optimizers? To answer this, we applied hyperparameter optimization to 120 SE data sets that explored bad smell detection, predicting Github ssue close time, bug report analysis, defect prediction, and dozens of other non-SE problems. We find that DODGE works best for data sets with low "intrinsic dimensionality" (D = 3) and very poorly for higher-dimensional data (D over 8). Nearly all the SE data seen here was intrinsically low-dimensional, indicating that DODGE is applicable for many SE Analytics tasks.

  • Better Software Analytics via “DUO”: Data mining algorithms using/used-by optimizers
    Empirical Software Engineering, 2020
    Co-Authors: Amritanshu Agrawal, Tim Menzies, Leandro L. Minku, Markus Wagner
    Abstract:

    This paper claims that a new field of empirical Software engineering research and practice is emerging: data mining using/used-by optimizers for empirical studies, or DUO . For example, data miners can generate models that are explored by optimizers. Also, optimizers can advise how to best adjust the control parameters of a data miner. This combined approach acts like an agent leaning over the shoulder of an analyst that advises “ask this question next” or “ignore that problem, it is not relevant to your goals”. Further, those agents can help us build “better” predictive models, where “better” can be either greater predictive accuracy or faster modeling time (which, in turn, enables the exploration of a wider range of options). We also caution that the era of papers that just use data miners is coming to an end. Results obtained from an unoptimized data miner can be quickly refuted, just by applying an optimizer to produce a different (and better performing) model. Our conclusion, hence, is that for Software Analytics it is possible, useful and necessary to combine data mining and optimization using DUO .

  • When, and Why, Simple Methods Fail. Lessons Learned from Hyperparameter Tuning in Software Analytics (and Elsewhere)
    2019
    Co-Authors: Amritanshu Agrawal, Tim Menzies
    Abstract:

    Tuning a data miner for for Software Analytics is something of a black art. Recent research has shown that some of that tuning can be achieved via automatic tools, called "hyperparameter optimizers". Much of that research has used tools developed from outside of SE. Hence, here, we ask how and when we can exploit the special properties of SE data to build faster and better optimizers. Specifically, we apply hyperparameter optimization for 120 data sets addressing problems like bad smell detection, predicting Github issue close time, bug report analysis, defect prediction and dozens of other non-SE problems. To these, we applied a tool developed using SE data which (a) out-performs the state-of-the-art for these SE problems yet (b) fails badly on non-SE problems. From this experience, we can infer a simple rule for when to use/avoid different kinds of optimizers. SE data is often about infrequent issues, like the occasional defect, or the rarely exploited security violation, or the requirement that holds for one special case. But as we show, the same was not observed when we applied it on non-SE data. Our conclusion will be that we can exploit this special properties of SE to great effect; specifically, to find better optimizations for SE tasks via a tactic called "dodging" (explained in this paper).

  • How to "DODGE" Complex Software Analytics?
    IEEE Transactions on Software Engineering, 2019
    Co-Authors: Amritanshu Agrawal, Xipeng Shen, Di Chen, Tim Menzies
    Abstract:

    Machine learning techniques applied to Software engineering tasks can be improved by hyperparameter optimization, i.e., automatic tools that find good settings for a learner's control parameters. We show that such hyperparameter optimization can be unnecessarily slow, particularly when the optimizers waste time exploring "redundant tunings", i.e., pairs of tunings which lead to indistinguishable results. By ignoring redundant tunings, DODGE, a tuning tool, runs orders of magnitude faster, while also generating learners with more accurate predictions than seen in prior state-of-the-art approaches.

  • ICSE (SEIP) - Take control: on the unreasonable effectiveness of Software Analytics
    2019 IEEE ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), 2019
    Co-Authors: Tim Menzies
    Abstract:

    The goal of Software Analytics should be insight; i.e. surprises that make us change the way we do business (in our case, the business of Software construction, maintenance and evolution). So what insights have we learned from Software Analytics? Our field is rife with any number of truisms that are commonly quoted but rarely checked. That stops now. Using Software Analytics, we can check, and prune, many of those truisms. Specifically, the number of variables required to make predictions about SE projects is remarkably small---which means that (a) most of the things we think might affect Software quality have little impact in practice; (b) controlling just a few key variables can be enough to improve Software quality.

Tao Xie - One of the best experts on this subject based on the ideXlab platform.

  • experience report on applying Software Analytics in incident management of online service
    Automated Software Engineering, 2017
    Co-Authors: Jianguang Lou, Dongmei Zhang, Qingwei Lin, Rui Ding, Tao Xie
    Abstract:

    As online services become more and more popular, incident management has become a critical task that aims to minimize the service downtime and to ensure high quality of the provided services. In practice, incident management is conducted through analyzing a huge amount of monitoring data collected at runtime of a service. Such data-driven incident management faces several significant challenges such as the large data scale, complex problem space, and incomplete knowledge. To address these challenges, we carried out 2-year Software-Analytics research where we designed a set of novel data-driven techniques and developed an industrial system called the Service Analysis Studio (SAS) targeting real scenarios in a large-scale online service of Microsoft. SAS has been deployed to worldwide product datacenters and widely used by on-call engineers for incident management. This paper shares our experience about using Software Analytics to solve engineers pain points in incident management, the developed data-analysis techniques, and the lessons learned from the process of research development and technology transfer.

  • Software Analytics and its application in practice
    Perspectives on Data Science for Software Engineering, 2016
    Co-Authors: Dongmei Zhang, Tao Xie
    Abstract:

    A huge wealth of data exists in Software life cycle, and hidden in the data is information about the quality of Software and services as well as the dynamics of Software development. With various analytical and computing technologies, Software Analytics is to obtain insightful and actionable information for data-driven tasks in engineering Software and services. In this chapter, we discuss the different aspects of Software Analytics, and we also share our lessons learned when putting Software Analytics into practice.

  • Perspectives on Data Science for Software Engineering - Software Analytics and its application in practice.
    Perspectives on Data Science for Software Engineering, 2016
    Co-Authors: Dongmei Zhang, Tao Xie
    Abstract:

    A huge wealth of data exists in Software life cycle, and hidden in the data is information about the quality of Software and services as well as the dynamics of Software development. With various analytical and computing technologies, Software Analytics is to obtain insightful and actionable information for data-driven tasks in engineering Software and services. In this chapter, we discuss the different aspects of Software Analytics, and we also share our lessons learned when putting Software Analytics into practice.

  • Software Analytics for incident management of online services an experience report
    Automated Software Engineering, 2013
    Co-Authors: Jianguang Lou, Dongmei Zhang, Qingwei Lin, Rui Ding, Tao Xie
    Abstract:

    As online services become more and more popular, incident management has become a critical task that aims to minimize the service downtime and to ensure high quality of the provided services. In practice, incident management is conducted through analyzing a huge amount of monitoring data collected at runtime of a service. Such data-driven incident management faces several significant challenges such as the large data scale, complex problem space, and incomplete knowledge. To address these challenges, we carried out two-year Software-Analytics research where we designed a set of novel data-driven techniques and developed an industrial system called the Service Analysis Studio (SAS) targeting real scenarios in a large-scale online service of Microsoft. SAS has been deployed to worldwide product datacenters and widely used by on-call engineers for incident management. This paper shares our experience about using Software Analytics to solve engineers' pain points in incident management, the developed data-analysis techniques, and the lessons learned from the process of research development and technology transfer.

  • Software Analytics achievements and challenges
    International Conference on Software Engineering, 2013
    Co-Authors: Dongmei Zhang, Tao Xie
    Abstract:

    A huge wealth of various data exist in the practice of Software development. Further rich data are produced by modern Software and services in operation, many of which tend to be data-driven and/or data-producing in nature. Hidden in the data is information about the quality of Software and services or the dynamics of Software development. Software Analytics is to utilize a data-driven approach to enable Software practitioners to perform data exploration and analysis in order to obtain insightful and actionable information; such information is used for completing various tasks around Software systems, Software users, and Software development process. This tutorial presents achievements and challenges of research and practice on principles, techniques, and applications of Software Analytics, highlighting success stories in industry, research achievements that are transferred to industrial practice, and future research and practice directions in Software Analytics.

Dongmei Zhang - One of the best experts on this subject based on the ideXlab platform.

  • experience report on applying Software Analytics in incident management of online service
    Automated Software Engineering, 2017
    Co-Authors: Jianguang Lou, Dongmei Zhang, Qingwei Lin, Rui Ding, Tao Xie
    Abstract:

    As online services become more and more popular, incident management has become a critical task that aims to minimize the service downtime and to ensure high quality of the provided services. In practice, incident management is conducted through analyzing a huge amount of monitoring data collected at runtime of a service. Such data-driven incident management faces several significant challenges such as the large data scale, complex problem space, and incomplete knowledge. To address these challenges, we carried out 2-year Software-Analytics research where we designed a set of novel data-driven techniques and developed an industrial system called the Service Analysis Studio (SAS) targeting real scenarios in a large-scale online service of Microsoft. SAS has been deployed to worldwide product datacenters and widely used by on-call engineers for incident management. This paper shares our experience about using Software Analytics to solve engineers pain points in incident management, the developed data-analysis techniques, and the lessons learned from the process of research development and technology transfer.

  • Perspectives on Data Science for Software Engineering - Software Analytics and its application in practice.
    Perspectives on Data Science for Software Engineering, 2016
    Co-Authors: Dongmei Zhang, Tao Xie
    Abstract:

    A huge wealth of data exists in Software life cycle, and hidden in the data is information about the quality of Software and services as well as the dynamics of Software development. With various analytical and computing technologies, Software Analytics is to obtain insightful and actionable information for data-driven tasks in engineering Software and services. In this chapter, we discuss the different aspects of Software Analytics, and we also share our lessons learned when putting Software Analytics into practice.

  • Software Analytics and its application in practice
    Perspectives on Data Science for Software Engineering, 2016
    Co-Authors: Dongmei Zhang, Tao Xie
    Abstract:

    A huge wealth of data exists in Software life cycle, and hidden in the data is information about the quality of Software and services as well as the dynamics of Software development. With various analytical and computing technologies, Software Analytics is to obtain insightful and actionable information for data-driven tasks in engineering Software and services. In this chapter, we discuss the different aspects of Software Analytics, and we also share our lessons learned when putting Software Analytics into practice.

  • Software Analytics for incident management of online services an experience report
    Automated Software Engineering, 2013
    Co-Authors: Jianguang Lou, Dongmei Zhang, Qingwei Lin, Rui Ding, Tao Xie
    Abstract:

    As online services become more and more popular, incident management has become a critical task that aims to minimize the service downtime and to ensure high quality of the provided services. In practice, incident management is conducted through analyzing a huge amount of monitoring data collected at runtime of a service. Such data-driven incident management faces several significant challenges such as the large data scale, complex problem space, and incomplete knowledge. To address these challenges, we carried out two-year Software-Analytics research where we designed a set of novel data-driven techniques and developed an industrial system called the Service Analysis Studio (SAS) targeting real scenarios in a large-scale online service of Microsoft. SAS has been deployed to worldwide product datacenters and widely used by on-call engineers for incident management. This paper shares our experience about using Software Analytics to solve engineers' pain points in incident management, the developed data-analysis techniques, and the lessons learned from the process of research development and technology transfer.

  • Software Analytics achievements and challenges
    International Conference on Software Engineering, 2013
    Co-Authors: Dongmei Zhang, Tao Xie
    Abstract:

    A huge wealth of various data exist in the practice of Software development. Further rich data are produced by modern Software and services in operation, many of which tend to be data-driven and/or data-producing in nature. Hidden in the data is information about the quality of Software and services or the dynamics of Software development. Software Analytics is to utilize a data-driven approach to enable Software practitioners to perform data exploration and analysis in order to obtain insightful and actionable information; such information is used for completing various tasks around Software systems, Software users, and Software development process. This tutorial presents achievements and challenges of research and practice on principles, techniques, and applications of Software Analytics, highlighting success stories in industry, research achievements that are transferred to industrial practice, and future research and practice directions in Software Analytics.

Xipeng Shen - One of the best experts on this subject based on the ideXlab platform.

  • Simpler Hyperparameter Optimization for Software Analytics: Why, How, When?
    arXiv: Software Engineering, 2020
    Co-Authors: Amritanshu Agrawal, Xueqi Yang, Rishabh Agrawal, Xipeng Shen, Tim Menzies
    Abstract:

    How to make Software Analytics simpler and faster? One method is to match the complexity of analysis to the intrinsic complexity of the data being explored. For example, hyperparameter optimizers find the control settings for data miners that improve for improving the predictions generated via Software Analytics. Sometimes, very fast hyperparameter optimization can be achieved by just DODGE-ing away from things tried before. But when is it wise to use DODGE and when must we use more complex (and much slower) optimizers? To answer this, we applied hyperparameter optimization to 120 SE data sets that explored bad smell detection, predicting Github ssue close time, bug report analysis, defect prediction, and dozens of other non-SE problems. We find that DODGE works best for data sets with low "intrinsic dimensionality" (D = 3) and very poorly for higher-dimensional data (D over 8). Nearly all the SE data seen here was intrinsically low-dimensional, indicating that DODGE is applicable for many SE Analytics tasks.

  • How to "DODGE" Complex Software Analytics?
    IEEE Transactions on Software Engineering, 2019
    Co-Authors: Amritanshu Agrawal, Xipeng Shen, Di Chen, Tim Menzies
    Abstract:

    Machine learning techniques applied to Software engineering tasks can be improved by hyperparameter optimization, i.e., automatic tools that find good settings for a learner's control parameters. We show that such hyperparameter optimization can be unnecessarily slow, particularly when the optimizers waste time exploring "redundant tunings", i.e., pairs of tunings which lead to indistinguishable results. By ignoring redundant tunings, DODGE, a tuning tool, runs orders of magnitude faster, while also generating learners with more accurate predictions than seen in prior state-of-the-art approaches.

  • Hyperparameter Optimization for Effort Estimation
    arXiv: Software Engineering, 2018
    Co-Authors: Tianpei Xia, Rahul Krishna, Xipeng Shen, Jianfeng Chen, George Mathew, Tim Menzies
    Abstract:

    Software Analytics has been widely used in Software engineering for many tasks such as generating effort estimates for Software projects. One of the "black arts" of Software Analytics is tuning the parameters controlling a data mining algorithm. Such hyperparameter optimization has been widely studied in other Software Analytics domains (e.g. defect prediction and text mining) but, so far, has not been extensively explored for effort estimation. Accordingly, this paper seeks simple, automatic, effective and fast methods for finding good tunings for automatic Software effort estimation. We introduce a hyperparameter optimization architecture called OIL (Optimized Inductive Learning). We test OIL on a wide range of hyperparameter optimizers using data from 945 Software projects. After tuning, large improvements in effort estimation accuracy were observed (measured in terms of standardized accuracy). From those results, we recommend using regression trees (CART) tuned by different evolution combine with default analogy-based estimator. This particular combination of learner and optimizers often achieves in a few hours what other optimizers need days to weeks of CPU time to accomplish. An important part of this analysis is its reproducibility and refutability. All our scripts and data are on-line. It is hoped that this paper will prompt and enable much more research on better methods to tune Software effort estimators.

  • Tuning for Software Analytics
    Information and Software Technology, 2016
    Co-Authors: Tim Menzies, Xipeng Shen
    Abstract:

    Context: Data miners have been widely used in Software engineering to, say, generate defect predictors from static code measures. Such static code defect predictors perform well compared to manual methods, and they are easy to use and useful to use. But one of the "black arts" of data mining is setting the tunings that control the miner.Objective: We seek simple, automatic, and very effective method for finding those tunings.Method: For each experiment with different data sets (from open source JAVA systems), we ran differential evolution as an optimizer to explore the tuning space (as a first step) then tested the tunings using hold-out data.Results: Contrary to our prior expectations, we found these tunings were remarkably simple: it only required tens, not thousands, of attempts to obtain very good results. For example, when learning Software defect predictors, this method can quickly find tunings that alter detection precision from 0% to 60%.Conclusion: Since (1)?the improvements are so large, and (2)?the tuning is so simple, we need to change standard methods in Software Analytics. At least for defect prediction, it is no longer enough to just run a data miner and present the result without conducting a tuning optimization study. The implication for other kinds of Analytics is now an open and pressing issue.

Michele Lanza - One of the best experts on this subject based on the ideXlab platform.

  • samoa a visual Software Analytics platform for mobile applications
    International Conference on Software Maintenance, 2013
    Co-Authors: Roberto Minelli, Michele Lanza
    Abstract:

    Mobile applications, also known as apps, are dedicated Software systems that run on handheld devices, such as smartphones and tablet computers. The apps business has in a few years turned into a multi-billion dollar market. From a Software engineering perspective apps represent a new phenomenon, and there is a need for tools and techniques to analyze apps. We present SAMOA, a visual web-based Software Analytics platform for mobile applications. It mines Software repositories of apps and uses a set of visualization techniques to present the mined data. We describe SAMOA, detail the analyses it supports, and describe a methodology to understand apps from a structural and historical perspective. The website of SAMOA, containing the screen cast of the tool demo, is located at http://samoa.inf.usi.ch/about.

  • Software Analytics for mobile applications insights a lessons learned
    Conference on Software Maintenance and Reengineering, 2013
    Co-Authors: Roberto Minelli, Michele Lanza
    Abstract:

    Mobile applications, known as apps, are Software systems running on handheld devices, such as smartphones and tablet PCs. The market of apps has rapidly expanded in the past few years into a multi-billion dollar business. Being a new phenomenon, it is unclear whether approaches to maintain and comprehend traditional Software systems can be ported to the context of apps. We present a novel approach to comprehend apps from a structural and historical perspective, leveraging three factors for the analysis: source code, usage of third-party APIs, and historical data. We implemented our approach in a web-based Software Analytics platform named SAMOA. We detail our approach and the supporting tool, and present a number of findings obtained while investigating a corpus of mobile applications. Our findings reveal that apps differ significantly from traditional Software systems in a number of ways, which calls for the development of novel approaches to maintain and comprehend them.

  • ICSM - SAMOA -- A Visual Software Analytics Platform for Mobile Applications
    2013 IEEE International Conference on Software Maintenance, 2013
    Co-Authors: Roberto Minelli, Michele Lanza
    Abstract:

    Mobile applications, also known as apps, are dedicated Software systems that run on handheld devices, such as smartphones and tablet computers. The apps business has in a few years turned into a multi-billion dollar market. From a Software engineering perspective apps represent a new phenomenon, and there is a need for tools and techniques to analyze apps. We present SAMOA, a visual web-based Software Analytics platform for mobile applications. It mines Software repositories of apps and uses a set of visualization techniques to present the mined data. We describe SAMOA, detail the analyses it supports, and describe a methodology to understand apps from a structural and historical perspective. The website of SAMOA, containing the screen cast of the tool demo, is located at http://samoa.inf.usi.ch/about.

  • CSMR - Software Analytics for Mobile Applications--Insights a Lessons Learned
    2013 17th European Conference on Software Maintenance and Reengineering, 2013
    Co-Authors: Roberto Minelli, Michele Lanza
    Abstract:

    Mobile applications, known as apps, are Software systems running on handheld devices, such as smartphones and tablet PCs. The market of apps has rapidly expanded in the past few years into a multi-billion dollar business. Being a new phenomenon, it is unclear whether approaches to maintain and comprehend traditional Software systems can be ported to the context of apps. We present a novel approach to comprehend apps from a structural and historical perspective, leveraging three factors for the analysis: source code, usage of third-party APIs, and historical data. We implemented our approach in a web-based Software Analytics platform named SAMOA. We detail our approach and the supporting tool, and present a number of findings obtained while investigating a corpus of mobile applications. Our findings reveal that apps differ significantly from traditional Software systems in a number of ways, which calls for the development of novel approaches to maintain and comprehend them.