Module reference¶
Overview¶
The scikitlearn library provides functionality for training linear models and a large number of related tools. The present module provides simplified interfaces for various linear model regression methods. These methods are set up in a way that work out of the box for typical problems in cluster expansion and force constant potential construction, including slight adjustments to scikitlearn default values.
The list of available linear regression method through the trainstation interface is
Ordinary LeastSquares (OLS)
LeastSquares with regularization matrix
Least Absolute Shrinkage and Selection Operator (LASSO)
AdaptiveLASSO
Ridge and Bayesianridge
Elasticnet
Recursive Feature Elimination (RFE)
Automatic Relevance Determination Regression (ARDR)
Fitting using Orthogonal Matching Pursuit (OMP)
L1regularization with splitBregman
The most commonly used fit methods in the present context are LASSO, automatic relevance determination regression (ARDR), recursive feature elimination with \(\ell_2\)fitting (RFEL2) as well as ordinary leastsquares optimization (OLS). Below follows a short summary of the main algorithms. More information about the available linear models can be found in the scikitlearn documentation.
Leastsquares¶
Ordinary leastsquares (OLS) optimization is providing a solution to the linear problem
where \(\boldsymbol{A}\) is the sensing matrix, \(\boldsymbol{y}\) is the vector of target values, and \(\boldsymbol{x}\) is the solution (parameter vector) that one seeks to obtain. The objective is given by
The OLS method is chosen by setting the fit_method
keyword to
leastsquares
.
Leastsquares with regularization matrix¶
Similar to OLS, leastsquares with regularization matrix optimization solves the linear problem
with the difference that in this case, an explicit regularization matrix \(\boldsymbol{\Lambda}\) is present such that the objective becomes
From a Bayesian perspective, the regularization matrix \(\boldsymbol{\Lambda}\) can be thought of as the inverse of the covariance matrix for the prior probability of the features [MueCed09]. It can be used to scale and couple features based on prior beliefs.
The leastsquares with regularization matrix method is chosen by setting the fit_method
keyword to leastsquareswithregmatrix
.
The regularization matrix is set via the reg_matrix
keyword.
If no matrix is specified, all elements will be set to zero and the OLS result is recovered.
Note that if standardization is used, the regularization matrix should still be designed with respect to the input sensing matrix and not the standardized on.
Parameter 
Type 
Description 
Default 



regularization matrix 

LASSO¶
The least absolute shrinkage and selection operator (LASSO) is a method for performing variable selection and regularization in problems in statistics and machine learning. The optimization objective is given by
While the first term ensures that \(\boldsymbol{x}\) is a solution to the linear problem at hand, the second term introduces regularization and guides the algorithm toward finding sparse solutions, in the spirit of compressive sensing. In general, LASSO is suited for solving strongly underdetermined problems.
The LASSO optimizer is chosen by setting the fit_method
keyword to
lasso
. The \(\alpha\) parameter is set via the alpha
keyword. If no
value is specified a line scan will be carried out automatically to determine
the optimal value.
Parameter 
Type 
Description 
Default 



controls the sparsity of the solution vector 

Automatic relevance determination regression (ARDR)¶
Automatic relevance determination regression (ARDR) is an optimization algorithm provided by scikitlearn that is similar to Bayesian Ridge Regression, which provides a probabilistic model of the regression problem at hand. The method is also known as Sparse Bayesian Learning and Relevance Vector Machine.
The ARDR optimizer is chosen by setting the fit_method
keyword to ardr
.
The threshold lambda parameter, which controls the sparsity of the solution
vector, is set via the threshold_lambda
keyword (default: 1e4).
Parameter 
Type 
Description 
Default 



controls the sparsity of the solution vector 

splitBregman¶
The splitBregman method [GolOsh09] is designed to solve a broad class of \(\ell_1\)regularized problems. The solution vector \(\boldsymbol{x}\) is given by
where \(\boldsymbol{d}\) is an auxiliary quantity, while \(\mu\) and \(\lambda\) are hyperparameters that control the sparseness of the solution and the efficiency of the algorithm.
The approach can be optimized by addition of a preconditioning step [ZhoSadAbe19].
This speedup enables efficient hyperparameter optimization of mu values.
By default, the splitbregman
fit method will trial a range of
\(\mu\) values and choose the optimal based on cross validation.
The splitBregman implementation supports the following keywords.
Parameter 
Type 
Description 
Default 



sparseness parameter 



weight of additional L2norm in splitBregman 



maximal number of splitBregman iterations 



convergence criterion of iterative minimization 



convergence criterion of conjugate gradient step 



maximal number of conjugate gradient iterations 



number of CV splits for finding optimal mu value 



how often to print fitting information to stdout 

Recursive feature elimination¶
Recursive feature elimination (RFE) is a feature selection algorithm that obtains the optimal features by carrying out a series of fits, starting with the full set of parameters and then iteratively eliminating the less important ones. RFE needs to be combined with a specific fit method. Since RFE may require many hundreds of single fits its often advisable to use ordinary leastsquares as training method, which is the default behavior. The present implementation is based on the implementation of feature selection in scikitlearn.
The RFE optimizer is chosen by setting the fit_method
keyword to
rfe
. The n_features
keyword allows one to specify the number of
features to select. If this parameter is left unspecified RFE with
crossvalidation will be used to determine the optimal number of features.
After the optimal number of features has been determined the final model is
trained. The fit method for the final fit can be controlled via
final_estimator
. Here, estimator
and final_estimator
can be set to
any of the fit methods described in this section. For example,
estimator='lasso'
implies that a LASSOCV scan is carried out for each fit
in the RFE algorithm.
Parameter 
Type 
Description 
Default 



number of features to select 



number parameters to eliminate 


percentage of parameters to eliminate 




number of CV splits (90/10) used when optimizing 



fit method to be used in RFE algorithm 



fit method to be used in the final fit 
= 


keyword arguments for fit method defined by 



keyword arguments for fit method defined by 

Note
When running on multicore systems please be mindful of memory
consumption. By default all CPUs will be used (n_jobs=1
), which
will duplicate data and can require a lot of memory, potentially
giving rise to errors. To prevent this behavior you can set the
n_jobs parameter
explicitly, which is handed over directly to scikitlearn.
Other methods¶
The optimizers furthermore support the ridge
method
(ridge
), the elastic net
method
(elasticnet
) as well as Bayesian ridge regression
(bayesianridge
).
Optimizer¶
 class trainstation.Optimizer(fit_data, fit_method='leastsquares', standardize=True, train_size=0.9, test_size=None, train_set=None, test_set=None, check_condition=True, seed=42, **kwargs)[source]¶
This optimizer finds a solution to the linear \(\boldsymbol{A}\boldsymbol{x}=\boldsymbol{y}\) problem.
One has to specify either train_size/test_size or train_set/test_set If either train_set or test_set (or both) is specified the fractions will be ignored.
Warning
Repeatedly setting up an
Optimizer
object and training without changing the seed for the random number generator will yield identical or correlated results, to avoid this please specify a different seed when setting up multipleOptimizer
instances. Parameters
fit_data (tuple(numpy.ndarray, numpy.ndarray)) – the first element of the tuple represents the fit matrix A (N, M array) while the second element represents the vector of target values y (N array); here N (=rows of A, elements of y) equals the number of target values and M (=columns of A) equals the number of parameters
fit_method (str) – method to be used for training; possible choice are “ardr”, “bayesianridge”, “elasticnet”, “lasso”, “leastsquares”, “omp”, “rfe”, “ridge”, “splitbregman”
standardize (bool) – if True the fit matrix and target values are standardized before fitting, meaning columns in the fit matrix and th target values are rescaled to have a standard deviation of 1.0.
train_size (float or int) – If float represents the fraction of fit_data (rows) to be used for training. If int, represents the absolute number of rows to be used for training.
test_size (float or int) – If float represents the fraction of fit_data (rows) to be used for testing. If int, represents the absolute number of rows to be used for testing.
train_set (tuple or list(int)) – indices of rows of A/y to be used for training
test_set (tuple or list(int)) – indices of rows of A/y to be used for testing
check_condition (bool) – if True the condition number will be checked (this can be sligthly more time consuming for larger matrices)
seed (int) – seed for pseudo random number generator
 scatter_data_train¶
target and predicted value for each row in the training set
 Type
ScatterData
 scatter_data_test¶
target and predicted value for each row in the test set
 Type
ScatterData
 property AIC: float¶
Akaike information criterion (AIC) for the model
 Return type
float
 property BIC: float¶
Bayesian information criterion (BIC) for the model
 Return type
float
 property fit_method: str¶
Fit method
 Return type
str
 get_contributions(A)¶
Returns the average contribution for each row of A to the predicted values from each element of the parameter vector.
 property n_nonzero_parameters: int¶
Number of nonzero parameters
 Return type
int
 property n_parameters: int¶
Number of parameters (=columns in A matrix)
 Return type
int
 property n_target_values: int¶
Number of target values (=rows in A matrix)
 Return type
int
 property parameters_norm: float¶
Norm of the parameter vector
 Return type
float
 property rmse_test: float¶
Root mean squared error for test set
 Return type
float
 property rmse_train: float¶
Root mean squared error for training set
 Return type
float
 property seed: int¶
Seed used to initialize pseudo random number generator
 Return type
int
 property standardize: bool¶
If True standardize the fit matrix before fitting
 Return type
bool
 property summary: Dict[str, Any]¶
Comprehensive information about the optimizer
 Return type
Dict
[str
,Any
]
 property test_fraction: float¶
Fraction of rows included in test set
 Return type
float
 property test_set: List[int]¶
Indices of rows included in the test set
 Return type
List
[int
]
 property test_size: int¶
Number of rows included in test set
 Return type
int
 property train_fraction: float¶
Fraction of rows included in training set
 Return type
float
 property train_set: List[int]¶
Indices of rows included in the training set
 Return type
List
[int
]
 property train_size: int¶
Number of rows included in training set
 Return type
int
 write_summary(fname)¶
Writes summary dict to file.
 Return type
None
EnsembleOptimizer¶
 class trainstation.EnsembleOptimizer(fit_data, fit_method='leastsquares', standardize=True, ensemble_size=50, train_size=1.0, bootstrap=True, check_condition=True, seed=42, **kwargs)[source]¶
The ensemble optimizer carries out a series of single optimization runs using the
Optimizer
class in order to solve the linear \(\boldsymbol{A}\boldsymbol{x} = \boldsymbol{y}\) problem. Subsequently, it provides access to various ensemble averaged quantities such as errors and parameters.Warning
Repeatedly setting up a
EnsembleOptimizer
and training without changing the seed for the random number generator will yield identical or correlated results, to avoid this please specify a different seed when setting up multipleEnsembleOptimizer
instances. Parameters
fit_data (tuple(numpy.ndarray, numpy.ndarray)) – the first element of the tuple represents the fit matrix A (N, M array) while the second element represents the vector of target values y (N array); here N (=rows of A, elements of y) equals the number of target values and M (=columns of A) equals the number of parameters
fit_method (str) – method to be used for training; possible choice are “ardr”, “bayesianridge”, “elasticnet”, “lasso”, “leastsquares”, “omp”, “rfe”, “ridge”, “splitbregman”
standardize (bool) – if True the fit matrix and target values are standardized before fitting, meaning columns in the fit matrix and th target values are rescaled to have a standard deviation of 1.0.
ensemble_size (int) – number of fits in the ensemble
train_size (float or int) – if float represents the fraction of fit_data (rows) to be used for training; if int, represents the absolute number of rows to be used for training
bootstrap (bool) – if True sampling will be carried out with replacement
check_condition (bool) – if True the condition number will be checked (this can be sligthly more time consuming for larger matrices)
seed (int) – seed for pseudo random number generator
 property bootstrap: bool¶
True if sampling is carried out with replacement
 Return type
bool
 property ensemble_size: int¶
Number of train rounds
 Return type
int
 property error_matrix: numpy.ndarray¶
Matrix of fit errors where N is the number of target values and M is the number of fits (i.e., the size of the ensemble)
 Return type
 property fit_method: str¶
Fit method
 Return type
str
 get_contributions(A)¶
Returns the average contribution for each row of A to the predicted values from each element of the parameter vector.
 property n_nonzero_parameters: int¶
Number of nonzero parameters
 Return type
int
 property n_parameters: int¶
Number of parameters (=columns in A matrix)
 Return type
int
 property n_target_values: int¶
Number of target values (=rows in A matrix)
 Return type
int
 property parameters_norm: float¶
Norm of the parameter vector
 Return type
float
 property parameters_splits: List[numpy.ndarray]¶
All parameters vectors in the ensemble
 Return type
List
[ndarray
]
 predict(A, return_std=False)[source]¶
Predicts data given an input matrix \(\boldsymbol{A}\), i.e., \(\boldsymbol{A}\boldsymbol{x}\), where \(\boldsymbol{x}\) is the vector of the fitted parameters. The method returns the vector of predicted values and optionally also the vector of standard deviations.
By using all parameter vectors in the ensemble a standard deviation of the prediction can be obtained.
 property rmse_test: float¶
Ensemble average of root mean squared error over test sets
 Return type
float
 property rmse_test_splits: numpy.ndarray¶
Root mean squared test errors obtained during for each fit in ensemble
 Return type
 property rmse_train: float¶
Ensemble average of root mean squared error over train sets
 Return type
float
 property rmse_train_splits: numpy.ndarray¶
Root mean squared train errors obtained during for each fit in ensemble
 Return type
 property seed: int¶
Seed used to initialize pseudo random number generator
 Return type
int
 property standardize: bool¶
If True standardize the fit matrix before fitting
 Return type
bool
 property summary: Dict[str, Any]¶
Comprehensive information about the optimizer
 Return type
Dict
[str
,Any
]
 train()[source]¶
Carries out ensemble training and construct the final model by averaging over all models in the ensemble.
 Return type
None
 property train_size: int¶
Number of rows included in train sets; note that this will be different from the number of unique rows if boostrapping
 Return type
int
 write_summary(fname)¶
Writes summary dict to file.
 Return type
None
CrossValidationEstimator¶
 class trainstation.CrossValidationEstimator(fit_data, fit_method='leastsquares', standardize=True, validation_method='kfold', n_splits=10, check_condition=True, seed=42, **kwargs)[source]¶
This class provides an optimizer with cross validation for solving the linear \(\boldsymbol{A}\boldsymbol{x} = \boldsymbol{y}\) problem. Crossvalidation (CV) scores are calculated by splitting the available reference data in multiple different ways. It also produces the finalized model (using the full input data) for which the CV score is an estimation of its performance.
Warning
Repeatedly setting up a
CrossValidationEstimator
and training without changing the seed for the random number generator will yield identical or correlated results, to avoid this please specify a different seed when setting up multipleCrossValidationEstimator
instances. Parameters
fit_data (tuple(numpy.ndarray, numpy.ndarray)) – the first element of the tuple represents the fit matrix A (N, M array) while the second element represents the vector of target values y (N array); here N (=rows of A, elements of y) equals the number of target values and M (=columns of A) equals the number of parameters
fit_method (str) – method to be used for training; possible choice are “ardr”, “bayesianridge”, “elasticnet”, “lasso”, “leastsquares”, “omp”, “rfe”, “ridge”, “splitbregman”
standardize (bool) – if True the fit matrix and target values are standardized before fitting, meaning columns in the fit matrix and th target values are rescaled to have a standard deviation of 1.0.
validation_method (str) – method to use for crossvalidation; possible choices are “shufflesplit”, “kfold”
n_splits (int) – number of times the fit data set will be split for the crossvalidation
check_condition (bool) – if True the condition number will be checked (this can be sligthly more time consuming for larger matrices)
seed (int) – seed for pseudo random number generator
 scatter_data_train¶
contains target and predicted values from each individual traininig set in the crossvalidation split
 Type
ScatterData
 scatter_data_validation¶
contains target and predicted values from each individual validation set in the crossvalidation split
 Type
ScatterData
 property AIC: float¶
Akaike information criterion (AIC) for the model
 Return type
float
 property BIC: float¶
Bayesian information criterion (BIC) for the model
 Return type
float
 property R2_validation: float¶
Average R2 score for validation sets
 Return type
float
 property fit_method: str¶
Fit method
 Return type
str
 get_contributions(A)¶
Returns the average contribution for each row of A to the predicted values from each element of the parameter vector.
 property n_nonzero_parameters: int¶
Number of nonzero parameters
 Return type
int
 property n_nonzero_parameters_splits: numpy.ndarray¶
Number of nonzero parameters for each split
 Return type
 property n_parameters: int¶
Number of parameters (=columns in A matrix)
 Return type
int
 property n_splits: int¶
Number of splits (folds) used for crossvalidation
 Return type
int
 property n_target_values: int¶
Number of target values (=rows in A matrix)
 Return type
int
 property parameters_norm: float¶
Norm of the parameter vector
 Return type
float
 property parameters_splits: numpy.ndarray¶
All parameters obtained during crossvalidation
 Return type
 property rmse_train: float¶
Average root mean squared training error obtained during crossvalidation
 Return type
float
 property rmse_train_final: float¶
Root mean squared error when using the full set of input data
 Return type
float
 property rmse_train_splits: numpy.ndarray¶
Root mean squared training errors obtained during crossvalidation
 Return type
 property rmse_validation: float¶
Average root mean squared crossvalidation error
 Return type
float
 property rmse_validation_splits: numpy.ndarray¶
Root mean squared validation errors obtained during crossvalidation
 Return type
 property seed: int¶
Seed used to initialize pseudo random number generator
 Return type
int
 property standardize: bool¶
If True standardize the fit matrix before fitting
 Return type
bool
 property summary: Dict[str, Any]¶
Comprehensive information about the optimizer
 Return type
Dict
[str
,Any
]
 property validation_method: str¶
Validation method name
 Return type
str
 write_summary(fname)¶
Writes summary dict to file.
 Return type
None