Analyzing the expected performance of any predictive model is a very important part of the model building process. When you build a model in LityxIQ, it will automatically compute a wide variety of performance metrics that are applicable to the model you have built. See https://support.lityxiq.com/289834-Performance-Analysis-Metrics-to-Analyze-Classification-Models and https://support.lityxiq.com/050028-Performance-Analysis-Metrics-to-Analyze-Numeric-Prediction-Models for more information on the metrics.
In addition, LityxIQ will use a validation methodology to compute the metrics that the user can control in the model settings (see https://support.lityxiq.com/325939-Model-Settings-Validation for much more information).
The result is that LityxIQ will report a close approximation of how the model will perform when used in a production deployment. Plus, LityxIQ will also report many qualitative results as well, which can be just as important as the quantitative measures of performance. The qualitative results, such as variable importance lists and transformation details, help ensure that you understand the inner workings of the model as best as possible, ensuring that it is not a black box.
To begin analyzing model performance, select the model and then click Evaluate & Explore -> Performance Analysis from the Selected Model menu or from the right click menu.
(Note, for additional information specific to unsupervised clustering models, see https://support.lityxiq.com/472334-Model-Performance-Analysis--for-Unsupervised-Clustering-Models, and for additional information specific to time series forecasting models, see https://support.lityxiq.com/838044-Model-Performance-Analysis--for-Forecasting--Time-Series-Models).
LityxIQ will open a window that allows you to dig deeply into the model's quantitative and qualitative performance. The options will be explained below.
Model/Version - Select the model and model version you would like to analyze. Note that you can select multiple models/versions, up to a maximum of 5.
Analysis Type - Select from a list of analyses that you can perform. Some of quantitative in nature, and some are qualitative. They are explained below.
- Performance Summary - this is the default view. It provides a detailed view of all quantitative performance metrics for the selected models, versions, and iterations. You can select more or fewer metrics to show using the Metrics to Analyze dropdown.
- Performance Details - this provides a more detailed view of performance metrics, showing results by model deciles (or percentiles, or other segmentations). Here, you will be able to view model lift (gains) chart comparisons, error rates by decile, and much more.
- Iteration List - this provides documentation of the detailed settings that were used for each iteration of the models selected.
- Variable Importance - this provides a list of all variables in each selected model and iteration, along with a relative factor showing you how important it was in each model. Because you can select multiple models and iterations, you can easily compare which variables were important (or not) across multiple models and algorithms. More information is here: https://support.lityxiq.com/353757-Performance-Analysis-Variable-Importance.
- ROC Curves - for binary models, this option will show you a comparison of ROC curves for all selected models and iterations. You can find more information about ROC curves here: https://support.lityxiq.com/364317-C-Statistic-and-ROC-Curves.
- Model Coefficients - for regression style models (e.g., linear and logistic regression), this will provide a detailed list of the regression coefficients and statistical significance levels (p-values).
- Categorical or Numeric Missing Values - this will show the replacement value that was used for each categorical or numeric variable in the model that had missing values in the dataset.
- Normalization - if you had selected the option to normalize predictor variables (or if LityxIQ determined it was useful to do it), you will be provided with the details of how the normalization was performed for each variable.
- Categorical/Numeric Binning - if you had selected the option to bin numeric or categorical variables in your model settings (or if LityxIQ determined it was necessary, such as for missing value categorization or because there were too many values of a categorical variable), these options will show you the details of how bins were created. Note that in the case of numeric bins, the information provided shows you the cutpoints of the bins, with the values shown being included in the upper bound of each bin.
- Pre-processing - here, LityxIQ will show you any variables that it removed from the modeling process during the pre-processing stage, and tell you the reason. For example, categorical values with only one unique value, having too many unique values, or having one value that is too dominant will be removed. If you had multi-collinearity processing set for the model, it will let you know variables that were removed for this reason. There are other potential reasons well that you may see a variable listed here.
- Decision Tree (Graphical/Tabular) - if you had built any tree-based models such as CART or CHAID, you will have the option to interactively view the tree in graphical or tabular format. The graphical decision tree is the same as that provided by LityxIQ's automated insights functionality. For documentation on navigating the visualization, see the Multivariate Segmentation section of the article: https://support.lityxiq.com/062674-Reviewing-Automated-Insights.
Chart/Table - some analysis types allow you to view the output in either a graphical chart format, or a tabular format.
Aggregation Level - this option will appear if you have selected the Performance Details analysis. You can decide at what level of detail you wish to see the results. The default is deciles (model scores grouped into 10 segments), but you can choose others as well, such as quintiles, vigintiles, or percentiles.
Iterations - all iterations that are available for the model(s) and version(s) selected in the Model/Version option will be shown here. Some analysis types will allow you to multi-select iterations to review output, while others will only allow one selection at a time.
Metrics to Analyze - depending on the Analysis Type, you will have the ability to select for which metrics you want to see output. For some analysis types such as Performance Summary, you can select many at once. For others, such as viewing Performance Details in a chart, you can only select one at a time. Note that this option is not available for analysis types that are providing qualitative information, such as "Pre-processing" or "Categorical missing values".
Go button - after changing your selections, click Go to see the output.