GAP (GradientBasedLearningForCAP) - Chapter 4: Skeletal Category of Smooth Maps

Construct the category of skeletal smooth maps. Objects in this category are the Euclidean spaces \mathbb{R}^n for non-negative integers n. Morphisms are smooth maps between these spaces, represented by a map function and its Jacobian matrix.

4.1-2 ObjectConstructor

Construct the object representing the Euclidean space \mathbb{R}^{n} in the category of skeletal smooth maps. Points in Euclidean spaces are represented as row vectors.

4.1-3 MorphismConstructor

Construct a smooth morphism from source to target using the given datum. The datum should be a pair [ map_func, jacobian_func ] where:

4.1-4 SmoothMap

4.2 Attributes

4.2-1 RankOfObject

The rank (dimension) of the Euclidean space represented by the object obj. For an object representing \mathbb{R}^n, this returns n.

4.2-2 Map

The underlying map function of a smooth morphism f. For a morphism f: \mathbb{R}^m \to \mathbb{R}^n, this is a function that takes a list of m elements and returns a list of n elements.

4.2-3 JacobianMatrix

The Jacobian matrix function of a smooth morphism f. For a morphism f: \mathbb{R}^m \to \mathbb{R}^n, this is a function that takes a list of m elements and returns an n \times m matrix representing the partial derivatives.

4.3 Operations

4.3-1 Eval

4.3-2 EvalJacobianMatrix

4.4 Available Smooth Maps

4.4-1 AvailableSkeletalSmoothMaps

4.4-2 DummyInputForAffineTransformation

Generate dummy input expressions for an affine transformation with m inputs and n outputs. Its length is (m+1)n + m. Additional arguments specify the weight string, bias string, and a logits string. For example, for m=2 and n=3, with weight string "w", bias string "b", and logits string "z", the output is [ w1_1, w2_1, b_1, w1_2, w2_2, b_2, w1_3, w2_3, b_3, z1, z2 ].

4.5 Supported CAP Operations

4.5-1 SkeletalSmoothMaps

4.5-2 Generate Documentation

gap> LoadPackage( "GradientBasedLearningForCAP", false );
true
gap> CAP_INTERNAL_GENERATE_DOCUMENTATION_FOR_CATEGORY_INSTANCES(
>     [
>         [ SkeletalSmoothMaps, "SkeletalSmoothMaps", 0 ],
>     ],
>     "GradientBasedLearningForCAP",
>     "SkeletalCategoryOfSmoothMaps.autogen.gd",
>     "Skeletal Category of Smooth Maps",
>     "Supported CAP Operations"
> );

4.6 Examples

In this example, we demonstrate the usage of the SkeletalCategoryOfSmoothMaps by constructing objects and morphisms, performing various operations, and utilizing built-in functions.

gap> Smooth := SkeletalCategoryOfSmoothMaps( );
SkeletalSmoothMaps
gap> Display( Smooth );
A CAP category with name SkeletalSmoothMaps:

49 primitive operations were used to derive 92 operations for this category wh\
ich algorithmically
* IsCartesianCategory
* IsLinearCategoryOverCommutativeRing
* IsSymmetricMonoidalCategory
and furthermore mathematically
* IsStrictMonoidalCategory
gap> R2 := ObjectConstructor( Smooth, 2 );
ℝ^2
gap> R2 = Smooth.2;
true
gap> r := RandomMorphism( R2, R2, 3 );
ℝ^2 -> ℝ^2
gap> Display( r );
ℝ^2 -> ℝ^2

‣ 0.766 * x1 ^ 1 + 0.234 * x2 ^ 2 + 1.
‣ 0.388 * x1 ^ 1 + 0.459 * x2 ^ 2 + 0.278
gap> f := MorphismConstructor( Smooth,
>         Smooth.2,
>         Pair(
>           x -> [ x[1] ^ 2 + Sin( x[2] ), Exp( x[1] ) + 3 * x[2] ],
>           x -> [ [ 2 * x[1], Cos( x[2] ) ], [ Exp( x[1] ), 3 ] ] ),
>         Smooth.2 );
ℝ^2 -> ℝ^2
gap> Display( f );
ℝ^2 -> ℝ^2

‣ x1 ^ 2 + Sin( x2 )
‣ Exp( x1 ) + 3 * x2
gap> dummy_input := CreateContextualVariables( [ "x1", "x2" ] );
[ x1, x2 ]
gap> Map( f )( dummy_input );
[ x1 ^ 2 + Sin( x2 ), Exp( x1 ) + 3 * x2 ]
gap> JacobianMatrix( f )( dummy_input );
[ [ 2 * x1, Cos( x2 ) ], [ Exp( x1 ), 3 ] ]
gap> x := [ 0.2, 0.3 ];
[ 0.2, 0.3 ]
gap> Map( f )( x );
[ 0.33552, 2.1214 ]
gap> JacobianMatrix( f )( x );
[ [ 0.4, 0.955336 ], [ 1.2214, 3 ] ]
gap> g := MorphismConstructor( Smooth,
>         Smooth.2,
>         Pair(
>           x -> [ 3 * x[1], Exp( x[2] ), x[1] ^ 3 + Log( x[2] ) ],
>           x -> [ [ 3, 0 ],
>                  [ 0, Exp( x[2] ) ],
>                  [ 3 * x[1] ^ 2, 1 / x[2] ] ] ),
>         Smooth.3 );
ℝ^2 -> ℝ^3
gap> Display( g );
ℝ^2 -> ℝ^3

‣ 3 * x1
‣ Exp( x2 )
‣ x1 ^ 3 + Log( x2 )
gap> Map( g )( dummy_input );
[ 3 * x1, Exp( x2 ), x1 ^ 3 + Log( x2 ) ]
gap> JacobianMatrix( g )( dummy_input );
[ [ 3, 0 ], [ 0, Exp( x2 ) ], [ 3 * x1 ^ 2, 1 / x2 ] ]
gap> h := PostCompose( g, f );
ℝ^2 -> ℝ^3
gap> Display( h );
ℝ^2 -> ℝ^3

‣ 3 * (x1 ^ 2 + Sin( x2 ))
‣ Exp( Exp( x1 ) + 3 * x2 )
‣ (x1 ^ 2 + Sin( x2 )) ^ 3 + Log( Exp( x1 ) + 3 * x2 )
gap> x;
[ 0.2, 0.3 ]
gap> Map( h )( x );
[ 1.00656, 8.34283, 0.789848 ]
gap> Eval( h, x );
[ 1.00656, 8.34283, 0.789848 ]
gap> Map( g )( Map( f )( x ) );
[ 1.00656, 8.34283, 0.789848 ]
gap> JacobianMatrix( h )( x );
[ [ 1.2, 2.86601 ], [ 10.19, 25.0285 ], [ 0.710841, 1.7368 ] ]
gap> EvalJacobianMatrix( h, x );
[ [ 1.2, 2.86601 ], [ 10.19, 25.0285 ], [ 0.710841, 1.7368 ] ]
gap> JacobianMatrix( g )( Map( f )( x ) ) * JacobianMatrix( f )( x );
[ [ 1.2, 2.86601 ], [ 10.19, 25.0285 ], [ 0.710841, 1.7368 ] ]
gap> IsCongruentForMorphisms( Smooth, f + f, 2 * f );
true
gap> IsCongruentForMorphisms( Smooth, 2 * f - f, f );
true
gap> s := SimplifyMorphism( h, infinity );
ℝ^2 -> ℝ^3
gap> Display( s );
ℝ^2 -> ℝ^3

‣ 3 * x1 ^ 2 + 3 * Sin( x2 )
‣ Exp( 3 * x2 + Exp( x1 ) )
‣ (x1 ^ 2 + Sin( x2 )) ^ 3 + Log( 3 * x2 + Exp( x1 ) )
gap> R2 := Smooth.( 2 );
ℝ^2
gap> R3 := Smooth.( 3 );
ℝ^3
gap> DirectProduct( R2, R3 );
ℝ^5
gap> p1 := ProjectionInFactorOfDirectProduct( [ R2, R3 ], 1 );
ℝ^5 -> ℝ^2
gap> Display( p1 );
ℝ^5 -> ℝ^2

‣ x1
‣ x2
gap> p2 := ProjectionInFactorOfDirectProduct( [ R2, R3 ], 2 );
ℝ^5 -> ℝ^3
gap> Display( p2 );
ℝ^5 -> ℝ^3

‣ x3
‣ x4
‣ x5
gap> u := UniversalMorphismIntoDirectProduct( Smooth, [ f, g ] );
ℝ^2 -> ℝ^5
gap> Display( u );
ℝ^2 -> ℝ^5

‣ x1 ^ 2 + Sin( x2 )
‣ Exp( x1 ) + 3 * x2
‣ 3 * x1
‣ Exp( x2 )
‣ x1 ^ 3 + Log( x2 )
gap> PreCompose( Smooth, u, p1 ) = f;
true
gap> PreCompose( Smooth, u, p2 ) = g;
true
gap> Display( f );
ℝ^2 -> ℝ^2

‣ x1 ^ 2 + Sin( x2 )
‣ Exp( x1 ) + 3 * x2
gap> Display( g );
ℝ^2 -> ℝ^3
‣ 3 * x1
‣ Exp( x2 )
‣ x1 ^ 3 + Log( x2 )
gap> d := DirectProductFunctorial( Smooth, [ f, g ] );
ℝ^4 -> ℝ^5
gap> Display( d );
ℝ^4 -> ℝ^5

‣ x1 ^ 2 + Sin( x2 )
‣ Exp( x1 ) + 3 * x2
‣ 3 * x3
‣ Exp( x4 )
‣ x3 ^ 3 + Log( x4 )
gap> Rf := ReverseDifferential( f );
ℝ^4 -> ℝ^2
gap> Display( Rf );
ℝ^4 -> ℝ^2

‣ x3 * (2 * x1) + x4 * Exp( x1 )
‣ x3 * Cos( x2 ) + x4 * 3
gap> Display( Smooth.Sqrt );
ℝ^1 -> ℝ^1

‣ Sqrt( x1 )
gap> Display( Smooth.Exp );
ℝ^1 -> ℝ^1

‣ Exp( x1 )
gap> Display( Smooth.Log );
ℝ^1 -> ℝ^1

‣ Log( x1 )
gap> Display( Smooth.Sin );
ℝ^1 -> ℝ^1

‣ Sin( x1 )
gap> Display( Smooth.Cos );
ℝ^1 -> ℝ^1

‣ Cos( x1 )
gap> Display( Smooth.Constant( [ 1.2, 3.4 ] ) );
ℝ^0 -> ℝ^2

‣ 1.2
‣ 3.4
gap> Display( Smooth.Zero( 2, 3 ) );
ℝ^2 -> ℝ^3

‣ 0
‣ 0
‣ 0
gap> Display( Smooth.IdFunc( 2 ) );
ℝ^2 -> ℝ^2

‣ x1
‣ x2
gap> Display( Smooth.Relu( 2 ) );
ℝ^2 -> ℝ^2

‣ Relu( x1 )
‣ Relu( x2 )
gap> Display( Smooth.Mul( 3 ) );
ℝ^3 -> ℝ^1

‣ x1 * x2 * x3
gap> Display( Smooth.Sum( 3 ) );
ℝ^3 -> ℝ^1

‣ x1 + x2 + x3
gap> Display( Smooth.Mean( 3 ) );
ℝ^3 -> ℝ^1

‣ (x1 + x2 + x3) / 3
gap> Display( Smooth.Variance( 3 ) );
ℝ^3 -> ℝ^1

‣ ((x1 - (x1 + x2 + x3) / 3) ^ 2 + (x2 - (x1 + x2 + x3) / 3) ^ 2
   + (x3 - (x1 + x2 + x3) / 3) ^ 2) / 3
gap> Display( Smooth.StandardDeviation( 3 ) );
ℝ^3 -> ℝ^1

‣ Sqrt( ((x1 - (x1 + x2 + x3) / 3) ^ 2 + (x2 - (x1 + x2 + x3) / 3) ^ 2
  + (x3 - (x1 + x2 + x3) / 3) ^ 2) / 3 )
gap> Display( Smooth.Power( 6 ) );
ℝ^1 -> ℝ^1

‣ x1 ^ 6
gap> Display( Smooth.PowerBase( 6 ) );
ℝ^1 -> ℝ^1

‣ 6 ^ x1
gap> Display( Smooth.Sigmoid( 2 ) );
ℝ^2 -> ℝ^2

‣ 1 / (1 + Exp( - x1 ))
‣ 1 / (1 + Exp( - x2 ))
gap> Display( Smooth.Softmax( 2 ) );
ℝ^2 -> ℝ^2

‣ Exp( x1 ) / (Exp( x1 ) + Exp( x2 ))
‣ Exp( x2 ) / (Exp( x1 ) + Exp( x2 ))
gap> Display( Smooth.QuadraticLoss( 2 ) );
ℝ^4 -> ℝ^1

‣ ((x1 - x3) ^ 2 + (x2 - x4) ^ 2) / 2
gap> Display( Smooth.BinaryCrossEntropyLoss( 1 ) );
ℝ^2 -> ℝ^1

‣ - (x2 * Log( x1 ) + (1 - x2) * Log( 1 - x1 ))
gap> Display( Smooth.SigmoidBinaryCrossEntropyLoss( 1 ) );
ℝ^2 -> ℝ^1

‣ Log( 1 + Exp( - x1 ) ) + (1 - x2) * x1
gap> Display( Smooth.CrossEntropyLoss( 3 ) );
ℝ^6 -> ℝ^1

‣ (- (Log( x1 ) * x4 + Log( x2 ) * x5 + Log( x3 ) * x6)) / 3
gap> Display( Smooth.SoftmaxCrossEntropyLoss( 3 ) );
ℝ^6 -> ℝ^1

‣ ((Log( Exp( x1 ) + Exp( x2 ) + Exp( x3 ) ) - x1) * x4
  + (Log( Exp( x1 ) + Exp( x2 ) + Exp( x3 ) ) - x2) * x5
  + (Log( Exp( x1 ) + Exp( x2 ) + Exp( x3 ) ) - x3) * x6) / 3

In this example, we illustrate the various convenience ways to construct smooth maps. For instance, the smooth map:

gap> Smooth := SkeletalCategoryOfSmoothMaps( );
SkeletalSmoothMaps
gap> dummy_input := CreateContextualVariables( [ "x1", "x2" ] );
[ x1, x2 ]
gap> g1 := MorphismConstructor( Smooth,
>         Smooth.2,
>         Pair(
>           x -> [ 3 * x[1], Exp( x[2] ), x[1] ^ 3 + Log( x[2] ) ],
>           x -> [ [ 3, 0 ],
>                  [ 0, Exp( x[2] ) ],
>                  [ 3 * x[1] ^ 2, 1 / x[2] ] ] ),
>         Smooth.3 );
ℝ^2 -> ℝ^3
gap> Display( g1 );
ℝ^2 -> ℝ^3
‣ 3 * x1
‣ Exp( x2 )
‣ x1 ^ 3 + Log( x2 )
gap> Map( g1 )( dummy_input );
[ 3 * x1, Exp( x2 ), x1 ^ 3 + Log( x2 ) ]
gap> JacobianMatrix( g1 )( dummy_input );
[ [ 3, 0 ], [ 0, Exp( x2 ) ], [ 3 * x1 ^ 2, 1 / x2 ] ]
gap> "# Use python to compute the Jacobian Matrix";;
gap> g2 := SmoothMap( Smooth,
>         Smooth.2,
>         x -> [ 3 * x[1], Exp( x[2] ), x[1] ^ 3 + Log( x[2] ) ],
>         Smooth.3,
>         true
>       );
ℝ^2 -> ℝ^3
gap> g1 = g2;
true
gap> "# Use python to compute the Jacobian Matrix";;
gap> g3 := SmoothMap( Smooth,
>         Smooth.2,
>         [ "3 * x1", "Exp( x2 )", "x1 ^ 3 + Log( x2 )" ],
>         Smooth.3,
>         true
>       );
ℝ^2 -> ℝ^3
gap> g3 = g1;
true
gap> "# Lazy evaluation of the Jacobian Matrix";;
gap> g4 := SmoothMap( Smooth,
>         Smooth.2,
>         [ "3 * x1", "Exp( x2 )", "x1 ^ 3 + Log( x2 )" ],
>         Smooth.3,
>         false
>       );
ℝ^2 -> ℝ^3
gap> g4 = g1;
true
gap> Map( g4 )( dummy_input );
[ 3 * x1, Exp( x2 ), x1 ^ 3 + Log( x2 ) ]
gap> J := JacobianMatrix( g4 )( [ 1, 2 ] );
[ [ Diff( [ "x1", "x2" ], "3 * x1", 1 )( [ 1, 2 ] ),
      Diff( [ "x1", "x2" ], "3 * x1", 2 )( [ 1, 2 ] ) ], 
  [ Diff( [ "x1", "x2" ], "Exp( x2 )", 1 )( [ 1, 2 ] ),
      Diff( [ "x1", "x2" ], "Exp( x2 )", 2 )( [ 1, 2 ] ) ], 
  [ Diff( [ "x1", "x2" ], "x1 ^ 3 + Log( x2 )", 1 )( [ 1, 2 ] ), 
      Diff( [ "x1", "x2" ], "x1 ^ 3 + Log( x2 )", 2 )( [ 1, 2 ] ) ] ]
gap> "# Evaluation uses python to compute the derivatives";;
gap> Eval( J[3][2] );
1/2
gap> Diff( [ "x1", "x2" ], "x1 ^ 3 + Log( x2 )", 2 )( [ 1, 2 ] );
1/2
gap> Eval( J );
[ [ 3, 0 ], [ 0, 7.38906 ], [ 3, 1/2 ] ]

In the following example, we demonstrate the construction and usage of an affine transformation morphism within the SkeletalCategoryOfSmoothMaps.

4 Skeletal Category of Smooth Maps

4.1 Constructors

4.1-1 SkeletalCategoryOfSmoothMaps