Introduction to Theano
Theano is a powerful open-source numerical computation library often utilized in machine learning and deep learning. Developed by the MILA lab at the Université de Montréal, Theano allows you to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays (tensors) efficiently. It bridges the gap between mathematical computation and hardware acceleration: Theano can seamlessly execute code on both CPUs and GPUs, offering high performance for computationally intensive tasks.
- Symbolic Differentiation: Theano can compute derivatives for mathematical functions automatically, a feature that’s indispensable for deep learning algorithms like gradient descent.
- Optimized Code: Theano optimizes computations by reordering operations and merging certain expressions, leading to faster runtime and reduced memory usage.
- GPU Acceleration: It offloads computations to GPUs, enabling faster matrix and tensor operations.
- Flexible Symbolic Graphs: Theano uses symbolic variables and graphs for computation, giving users complete control over the data flow.
- Compatible with NumPy: Many operations in Theano mirror NumPy, making it easy to learn for those already familiar with Python’s scientific stack.
Although development of Theano has stopped as of 2017, it remains widely used in educational and academic contexts, and serves as the foundation for many higher-level libraries like Keras.
In this blog, we’ll dive into some core Theano functionalities, explore several APIs with examples, and implement a simple application to see Theano in action.
Top Theano APIs with Explanations and Code Snippets
Below is a comprehensive reference of Theano APIs, along with step-by-step examples for each.
1. Creating Symbolic Variables
Theano allows you to define symbolic variables for scalars, vectors, matrices, and tensors.
import theano
import theano.tensor as T
# Creating scalar variables
x = T.scalar('x') # Symbolic scalar
y = T.scalar('y') # Symbolic scalar
# Creating vector and matrix variables
v = T.vector('v') # 1D array
A = T.matrix('A') # 2D array
print(x, v, A) # Displays symbolic variable information
2. Symbolic Mathematical Expressions
You can define symbolic expressions involving addition, subtraction, multiplication, division, and more.
# Symbolic addition and multiplication z = x + y w = x * y # Define a function to compute this expression f = theano.function([x, y], [z, w]) print(f(2, 3)) # Output: [5, 6]
3. Applying Activation Functions
Many deep learning problems require applying activation functions like sigmoid, tanh, or ReLU to data. Theano supports these functions natively.
# Symbolic sigmoid and tanh functions sigmoid = T.nnet.sigmoid(x) tanh = T.tanh(x) # Define a computation function activation_fn = theano.function([x], [sigmoid, tanh]) print(activation_fn(0)) # Output: [0.5, 0.0]
4. Automatic Gradient Computation
One of Theano’s key features is its ability to compute symbolic gradients for optimization.
# Define a quadratic expression y = x**2 + 3*x + 5 # Compute the gradient of y w.r.t. x dy_dx = T.grad(y, x) gradient_fn = theano.function([x], dy_dx) print(gradient_fn(2)) # Output: 7 (gradient at x=2)
5. Shared Variables
Shared variables provide a way to store parameters and retain their values across function calls, making them useful for machine learning models.
# Creating a shared variable w = theano.shared(5.0, name='w') # Define an expression with the shared variable z = w * x # Update the shared variable update = w + 1 update_fn = theano.function([], updates=[(w, update)]) print(w.get_value()) # Output: 5.0 update_fn() print(w.get_value()) # Output: 6.0
6. Using Scalars and Indexing
You can extract specific values from tensors or perform advanced indexing.
# Creating a 1D tensor
vector = T.vector('vector')
# Extract first element
first_elem = vector[0]
index_fn = theano.function([vector], first_elem)
print(index_fn([10, 20, 30])) # Output: 10
7. Dot Products and Matrix Multiplication
Linear algebra is a breeze with Theano. You can easily compute dot products or matrix multiplications.
# Symbolic vectors and matrices
v1 = T.vector('v1')
v2 = T.vector('v2')
m1 = T.matrix('m1')
# Dot product
dot = T.dot(v1, v2)
# Matrix multiplication
mat_mul = T.dot(m1, v1)
dot_fn = theano.function([v1, v2], dot)
mat_mul_fn = theano.function([m1, v1], mat_mul)
print(dot_fn([1, 2], [3, 4])) # Output: 11
print(mat_mul_fn([[1, 2], [3, 4]], [1, 0])) # Output: [1 3]
8. Element-wise Operations
Performing operations across array elements is straightforward.
# Element-wise addition and multiplication result = v1 + v2 elem_op_fn = theano.function([v1, v2], result) print(elem_op_fn([1, 2, 3], [4, 5, 6])) # Output: [5, 7, 9]
9. If-Then-Else Logic
Conditional operations in Theano can be specified using theano.tensor.switch.
# Define a condition condition = T.gt(x, 0) # x > 0 # If-else operation result = T.switch(condition, x**2, -x) cond_fn = theano.function([x], result) print(cond_fn(2)) # Output: 4 print(cond_fn(-3)) # Output: 3
10. Softmax Activation
Theano includes a built-in function for softmax, commonly used in classification problems.
probs = T.nnet.softmax(v) # Softmax over vector softmax_fn = theano.function([v], probs) print(softmax_fn([1, 2, 3])) # Output: [0.09, 0.24, 0.67]
11. Random Number Generation
Theano supports symbolic random number generation via RandomStreams.
srng = T.shared_randomstreams.RandomStreams(seed=42) random_val = srng.normal((2, 2)) # Generate a 2x2 normal distribution matrix rand_fn = theano.function([], random_val) print(rand_fn())
12. Cross-Entropy Loss
Used in classification, cross-entropy loss is implemented via Theano.
x = T.matrix('x')
y = T.ivector('y') # Class labels as integers
loss = T.nnet.categorical_crossentropy(x, y)
loss_fn = theano.function([x, y], loss)
print(loss_fn([[0.9, 0.1], [0.2, 0.8]], [0, 1]))
Example Application: Logistic Regression with Theano
Here’s how you can implement logistic regression on a toy dataset using Theano APIs.
import numpy as np
import theano
import theano.tensor as T
# Generating toy data
X_data = np.random.randn(100, 2)
y_data = (X_data[:, 0] * 0.5 + X_data[:, 1] > 0).astype(int)
# Symbolic variables
X = T.matrix('X') # Input feature matrix
y = T.ivector('y') # Labels
weights = theano.shared(np.random.randn(2).astype(theano.config.floatX), name='weights')
bias = theano.shared(0.0, name='bias')
# Logistic regression model
z = T.dot(X, weights) + bias
predictions = T.nnet.sigmoid(z)
# Loss: Cross-entropy
loss = T.mean(T.nnet.binary_crossentropy(predictions, y))
gradient_w = T.grad(loss, weights)
gradient_b = T.grad(loss, bias)
# Update rules
updates = [(weights, weights - 0.1 * gradient_w), (bias, bias - 0.1 * gradient_b)]
train_fn = theano.function([X, y], loss, updates=updates)
predict_fn = theano.function([X], predictions)
# Train model
for epoch in range(500):
loss_val = train_fn(X_data, y_data)
if epoch % 50 == 0:
print(f'Epoch {epoch}, Loss: {loss_val}')
# Test Predictions
test_preds = predict_fn(X_data[:5])
print("Predictions:", test_preds)
Conclusion
Theano might no longer be actively developed, but its contribution to the world of machine learning and symbolic computation has been monumental. From automatic differentiation to GPU acceleration and flexible computation graphs, it provides a solid framework for building custom ML systems. By leveraging Theano’s API, you can build innovative solutions tailored to your specific use cases.
Happy coding with Theano! 🚀
