MC++
Eigenvalue Arithmetic for Factorable Functions
Author
Nikola Perić, Akshay Shah, Jai Rajyaguru & Benoît Chachuat

Given a factorable, multivariate function \(f:\mathbb{R}^n\to\mathbb{R}\), that is twice-continuously differentiable on a box \(X:=[x^{\rm L},x^{\rm U}]\), Mönnigmann's technique [Mönnigmann, 2008; 2011] provides a means for computing spectral bounds of its Hessian matrix \(H_f(x)\) at any point \(x\in X\)—without actually computing \(H_f(x)\). Applications of this technique are in determining whether a function is convex or concave on a particular domain, as well as in constructing convex/concave relaxations for complete search approaches in global optimization. Alternative techniques for determining spectral bounds include the interval variant of Gershgorin's circle criterion [Adjiman et al., 1998] as well as Hertz & Rohn's method [Hertz, 1992], which rely on an interval enclosure of the set of all possible Hessian matrices \([H_f] \supseteq \{H_f(x) \mid x\in X\}\). See also McCormick Relaxation Arithmetic for Factorable Functions for an alternative way of constructing convex/concave relaxations.

The class mc::Specbnd provides an implementation of eigenvalue arithmetic. We note that mc::SpecBnd is not a verified implementation in the sense that rounding errors are not accounted for in propagating the spectral bounds.

The implementation of mc::Specbnd relies on the operator/function overloading mechanism of C++. This makes the computation of spectral bounds both simple and intuitive, similar to computing function values in real arithmetics. Moreover, mc::Specbnd can be used as the template parameter of other available types in MC++; for instance, mc::Specbnd can be used in order to propagate spectral bounds on the remainder term of Taylor variable mc::TVar. Likewise, mc::Specbnd can be used as the template parameter of the types fadbad::F, fadbad::B and fadbad::T of FADBAD++ for computing spectral bounds of either the partial derivatives or the Taylor coefficients of a factorable function too (see How do I compute spectral bounds of the partial derivatives or the Taylor coefficients of a factorable function using FADBAD++?).

mc:Specbnd is itself templated in the type used to propagate the necessary bounds. By default, mc::Specbnd can be used with the non-verified interval type mc::Interval of MC++. For reliability, however, it is strongly recommended to use verified interval arithmetic such as PROFIL (header file mcprofil.hpp) or FILIB++ (header file mcfilib.hpp). The types mc::McCormick and mc::TVar can also be used as template parameters of mc::Specbnd, thereby making it possible to compute, respectively, convex/concave bounds and Taylor models of the spectrum of \(H_f\).

As well as propagating spectral bounds for factorable functions using Mönnigmann's technique, mc::Specbnd provides support for computing spectral bounds based on the interval Hessian matrix of a twice continuously-differentiable, factorable function using either Gershgorin's circle criterion or Hertz & Rohn's method. As established by Darup et al. [2012], the spectral bound arithmetic may or may not produce tighter spectral bounds than these latter techniques, depending on the factorable function \(f\) at hand and its variable range \(X\).

Results obtained for the factorable function \(f(x,y)=1+x-\sin(2x+3y)-\cos(3x-5y)\) for \(x\in [-0.5,0.5]^2\) are shown in the figure below. This function (red line) and corresponding interval bounds (blue line) are shown in the left plot, while the minimal and maximal eigenvalues (red and green lines) as well as the computed spectral bounds (blue line) of the Hessian matrix \(H_f\) are shown on the right plot.

SB-2D_function.png
SB-2D_spectral.png

How do I compute (interval) bounds on the spectrum of the Hessian matrix of a factorable function?

Suppose we want to compute a spectral (interval) bound for the Hessian matrix of the real-valued function \(f(x_1,x_2,x_3)=\exp(x_1-2x_2^2+3x_3^3)\) for \((x_1,x_2,x_3)\in [-0.3,0.2]\times[-0.1,0.6]\times[-0.4,0.5]\). For simplicity, this bound is calculated using the default interval type, mc::Interval:

#include "interval.hpp"
#include "specbnd.hpp"
typedef mc::Interval I;
typedef mc::Specbnd<I> SB;

First, the variables \(x_1\), \(x_2\) and \(x_3\) are defined as follows:

SB X1( I(-0.3,0.2), 0, 3 );
SB X2( I(-0.1,0.6), 1, 3 );
SB X3( I(-0.4,0.5), 2, 3 );

Essentially, the first line means that X1 is a variable of class mc::Specbnd, belongs to the interval \([-0.3,0.2]\), and having index 0 out of 3 independent variables (recall that indexing in C/C++ starts at 0 by convention!). The same holds for the mc::Specbnd variable X2 and X3.

Having defined the variables, spectral bounds on the Hessian matrix \(H_f\) of \(f\) on \([-0.3,0.2]\times[-0.1,0.6]\times[-0.4,0.5]\) are simply calculated as:

SB F = exp( X1 - 2*sqr(X2) + 3*pow(X3,3) );

The computed spectral bounds can be retrieved as:

I specF = F.SI();

The function and first-derivative bounds can also be retrieved using mc::Specbnd::I and mc::Specbnd::FI. The results of the spectral bound propagation can be displayed to the standard output as:

std::cout << F << std::endl;

which produces the following display:

  [ -1.99039e+01 :  3.70043e+01 ]
  [  2.97601e-01 :  1.77713e+00 ]
  ( [  2.97601e-01 :  1.77713e+00 ] )
  ( [ -4.26511e+00 :  7.10852e-01 ] )
  ( [  0.00000e+00 :  3.99854e+00 ] )

Note that the display is organized as follows:

\begin{align*} & [\underline{\lambda}:\overline{\lambda}]\\ & [\underline{f}:\overline{f}]\\ & ( [\underline{\partial_{x_1}f}:\overline{\partial_{x_1}f}] )\\ & ( [\underline{\partial_{x_2}f}:\overline{\partial_{x_2}f}] )\\ & ( [\underline{\partial_{x_3}f}:\overline{\partial_{x_3}f}] ) \end{align*}

That is, the spectral bound is \([-19.904,37.004]\) here.

As noted earlier, other methods are available for bounding the spectrum of interval Hessian matrices, such as Gershgorin's circle criterion and Hertz & Rohn's method. mc::Specbnd also provides a means for computing these bounds, e.g. for comparison with the Mönnigmann's eigenvalue arithmetic. Interval Hessian matrices can be computed using the forward and/or reverse AD types of FADBAD++:

#include "mcfadbad.hpp"
typedef fadbad::F<I> FI;
typedef fadbad::F<FI> FFI;
typedef fadbad::B<FI> BFI;

Then, in order to compute spectral bounds for \(f(x_1,x_2,x_3)=\exp(x_1-2x_2^2+3x_3^3)\) on \((x_1,x_2,x_3)\in [-0.3,0.2]\times[-0.1,0.6]\times[-0.4,0.5]\), we proceed as follows:

FI FX1 = I(-0.3,0.2); FX1.diff(0,3);
FI FX2 = I(-0.1,0.6); FX2.diff(1,3);
FI FX3 = I(-0.4,0.5); FX3.diff(2,3);
FFI FFX1 = FX1; FFX1.diff(0,3);
FFI FFX2 = FX2; FFX2.diff(1,3);
FFI FFX3 = FX3; FFX3.diff(2,3);
FFI FFZ = exp( FFX1 - 2*pow(FFX2,2) + 3*pow(FFX3,3) );
std::pair<double,double> spbndF;
SB::options.HESSBND = SB::Options::GERSHGORIN;
spbndF = SB::spectral_bound( FFZ );
std::cout << "Spectral bound (Gershgorin, forward-forward): " << I(spbndF.first,spbndF.second) << std::endl;
SB::options.HESSBND = SB::Options::HERTZROHN;
spbndF = SB::spectral_bound( FFZ );
std::cout << "Spectral bound (Hertz&Rohn, forward-forward): " << I(spbndF.first,spbndF.second) << std::endl;

producing:

Spectral bound (Gershgorin, forward-forward): [ -2.63904e+01 :  3.85859e+01 ]
Spectral bound (Hertz&Rohn, forward-forward): [ -2.11973e+01 :  3.05272e+01 ]

In this case, the eigenvalue arithmetic provides tighter bounds than with Gershgorin's circle criterion, yet looser than with Herz & Rohn's method.

Spectral bounds for interval Hessian matrices generated with the forward-reverse mode of AD can also be computed:

BFI BFX[3] = { FX1, FX2, FX3 };
BFI BFZ = exp( BFX[0] - 2*pow(BFX[1],2) + 3*pow(BFX[2],3) );
BFZ.diff(0,1);
std::pair<double,double> spbndB;
SB::options.HESSBND = SB::Options::GERSHGORIN;
spbndB = SB::spectral_bound( BFX );
std::cout << "Spectral bound (Gershgorin, forward-reverse): " << I(spbndB.first,spbndB.second) << std::endl;
SB::options.HESSBND = SB::Options::HERTZROHN;
spbndB = SB::spectral_bound( BFX );
std::cout << "Spectral bound (Hertz&Rohn, forward-reverse): " << I(spbndB.first,spbndB.second) << std::endl;

producing:

Spectral bound (Gershgorin, forward-reverse): [ -2.63904e+01 :  3.85859e+01 ]
Spectral bound (Hertz&Rohn, forward-reverse): [ -2.11973e+01 :  3.05272e+01 ]

In this case, the spectral bounds computed from the Hessian matrix obtained with the forward-forward and forward-reverse modes of AD are indeed identical, although such may not always be the case; see [Darup et al., 2012]

How do I compute convex/concave relaxations on the spectrum of the Hessian matrix of a factorable function?

Instead of using standard interval arithmetic for propagating spectral bounds, we can as well make use of the McCormick relaxation technique to propagate convex/concave bounds. Using MC++, this is done simply by selecting mc::McCormick as the template parameter in mc::Specbnd:

#include "mccormick.hpp"
typedef mc::McCormick<I> MC;
typedef mc::Specbnd<MC> SBMC;

Then, the procedure for computing a spectral bound remains essentially the same as described in the previous section. In order to compute convex/concave spectral relaxations and subgradients at point \((0,0,0)\) of \(f(x_1,x_2,x_3)=\exp(x_1-2x_2^2+3x_3^3)\) for \((x_1,x_2,x_3)\in [-0.3,0.2]\times[-0.1,0.6]\times[-0.4,0.5]\), we proceed as follows:

SBMC X1( MC(I(-0.3,0.2),0.).sub(3,0), 0, 3 );
SBMC X2( MC(I(-0.1,0.6),0.).sub(3,1), 1, 3 );
SBMC X3( MC(I(-0.4,0.5),0.).sub(3,2), 2, 3 );
SBMC F = exp( X1 - 2*sqr(X2) + 3*pow(X3,3) );
MC specF = F.SI();
std::cout << F << std::endl;

The only difference here concerns the initialization of the McCormick variables inside the variables X and Y, which passes the bounds for the variable as well as the point at which the convex/concave bounds and their subgradients are computed—See How do I compute McCormick relaxations of a factorable function?

More information on the spectral relaxations can again be obtained as:

std::cout << F << std::endl;

which displays:

  [ -1.99039e+01 :  3.70043e+01 ] [ -1.54413e+01 :  2.81720e+01 ] [ (-9.27293e+00, 0.00000e+00,-5.21602e+00) : ( 1.72398e+01, 0.00000e+00, 1.50542e+01) ]
  [  2.97601e-01 :  1.77713e+00 ] [  8.45354e-01 :  1.37868e+00 ] [ ( 8.45354e-01,-8.45354e-01, 3.04327e-01) : ( 8.27940e-01, 0.00000e+00, 4.65716e-01) ]
  ( [  2.97601e-01 :  1.77713e+00 ] [  8.45354e-01 :  1.37868e+00 ] [ ( 8.45354e-01,-8.45354e-01, 3.04327e-01) : ( 8.27940e-01, 0.00000e+00, 4.65716e-01) ] )
  ( [ -4.26511e+00 :  7.10852e-01 ] [ -3.72711e-01 :  4.32433e-01 ] [ ( 3.38142e-01,-7.44666e+00, 1.21731e-01) : ( 3.31176e-01,-1.19041e+00, 1.86287e-01) ] )
  ( [  0.00000e+00 :  3.99854e+00 ] [  0.00000e+00 :  2.96812e+00 ] [ ( 0.00000e+00, 0.00000e+00, 0.00000e+00) : ( 1.86287e+00, 0.00000e+00, 1.31570e+00) ] )

By construction, the underlying interval bounds of the McCormick relaxations are identical to their spectral interval bound counterparts. The convex/concave bounds at \((0,0,0)\) come as additional information and are seen to be tighter than the interval bounds. The corresponding subgradients can also be used to construct affine relaxations.

Similarly, a Taylor model of the spectrum can be conveniently computed by selecting mc::TVar as the template parameter in mc::Specbnd.

Which functions are overloaded in mc::Specbnd eigenvalue arithmetic?

mc::TVar overloads the usual functions exp, log, sqr, sqrt, pow, inv, cos, sin, tan, acos, asin, atan. Unlike mc::Interval and mc::McCormick, the functions min, max and fabs are not overloaded in mc::TVar since they are not twice continuously differentiable. Also, erf and erfc are overloaded in mc::Specbnd but currently cannot be used due to a limitation in the third-party libarary FADBAD++, where these functions are not (yet?) overloaded.

How do I compute spectral bounds of the partial derivatives or the Taylor coefficients of a factorable function using FADBAD++?

The combination of mc::Specbnd with the classes fadbad::F, and fadbad::B of FADBAD++ to compute a spectral bound for the Hessian matrix of either the partial derivatives or the Taylor coefficients of a factorable function is essentially the same as with mc::McCormick (see How do I compute McCormick relaxations of the partial derivatives or the Taylor coefficients of a factorable function using FADBAD++?) or mc::TVar (see How do I compute Taylor models of the partial derivatives or the Taylor coefficients of a factorable function using FADBAD++?).

Next, we present the case of fadbad::F only. Continuing the previous example, spectral bounds of the partial derivatvies of \(f(x_1,x_2,x_3)=\exp(x_1-2x_2^2+3x_3^3)\) for \((x_1,x_2,x_3)\in [-0.3,0.2]\times[-0.1,0.6]\times[-0.4,0.5]\) can be computed as follows:

typedef fadbad::F<SB> FSB;
FSB FSBX1 = X1; FSBX1.diff(0,3);
FSB FSBX2 = X2; FSBX2.diff(1,3);
FSB FSBX3 = X3; FSBX3.diff(2,3);
FSB FSBF = exp( FSBX1 - 2*pow(FSBX2,2) + 3*pow(FSBX3,3) );
std::cout << "Spectral bounds of df/dx1:\n" << FSBF.d(0) << std::endl;
std::cout << "Spectral bounds of df/dx2:\n" << FSBF.d(1) << std::endl;
std::cout << "Spectral bounds of df/dx3:\n" << FSBF.d(2) << std::endl;

producing the output:

Spectral bounds of df/dx1:
  [ -1.99039e+01 :  3.70043e+01 ]
  [  2.97601e-01 :  1.77713e+00 ]
  ( [  2.97601e-01 :  1.77713e+00 ] )
  ( [ -4.26511e+00 :  7.10852e-01 ] )
  ( [  0.00000e+00 :  3.99854e+00 ] )

Spectral bounds of df/dx2:
  [ -1.16096e+02 :  8.92716e+01 ]
  [ -4.26511e+00 :  7.10852e-01 ]
  ( [ -4.26511e+00 :  7.10852e-01 ] )
  ( [ -8.81457e+00 :  9.04587e+00 ] )
  ( [ -9.59650e+00 :  1.59942e+00 ] )

Spectral bounds of df/dx3:
  [ -1.28567e+02 :  2.06229e+02 ]
  [  0.00000e+00 :  3.99854e+00 ]
  ( [  0.00000e+00 :  3.99854e+00 ] )
  ( [ -9.59650e+00 :  1.59942e+00 ] )
  ( [ -1.27953e+01 :  2.49909e+01 ] )

How are the options set for the computation of a spectral bound?

The class mc::Specbnd has a public static member called mc::Specbnd::options that can be used to set/modify the options; e.g.,

The available options are the following:

Options in mc::Specbnd::Options: name, type and description
Name TypeDefault Description
HESSBND mc::Specbnd::Options::HESSBND_STRATEGY mc::Specbnd::Options::GERSHGORIN Strategy for computing spectral bounds in interval Hessian matrix using mc::Specbnd::spectral_bound

Errors What errors can I encounter during computation of a spectral bound?

Errors are managed based on the exception handling mechanism of the C++ language. Each time an error is encountered, a class object of type mc::Specbnd::Exceptions is thrown, which contains the type of error. It is the user's responsibility to test whether an exception was thrown during the computation of a spectral bound, and then make the appropriate changes. Should an exception be thrown and not caught by the calling program, the execution will abort.

Possible errors encountered during the computation of a spectral bound are:

Errors during the Computation of a Spectral Bound
Number Description
1 Failed to compute spectrum in Specbnd::spectrum
2 Failed to compute spectral bound in Specbnd::spectral_bound
-1 Operation between variables with different numbers of dependents
-33 Feature not yet implemented in mc::Specbnd

Moreover, exceptions may be thrown by the template parameter class itself.

References