Skip to content

Commit 3326ee5

Browse files
carlportzcarlportz
committed
Enhance Deep Learning chapter by refining the introduction to neural networks and adding a new notebook on single-layer neural networks, including theoretical foundations and implementation details.
1 parent cdca273 commit 3326ee5

2 files changed

Lines changed: 159 additions & 2 deletions

File tree

‎chapters/deep_learning.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,9 +4,9 @@ downloads: []
44

55
# Deep Learning
66

7-
We dedicate the last chapter of this programming course to *neural networks*, a class of models that enabled rapid progress in machine learning applications and shifted *AI* from a niche research topic to a major technology driver, including applications in chemistry.
7+
We dedicate the last chapter of this programming course to *neural networks*, a class of models that enabled rapid progress in machine learning applications. Recent progress in training behaviour and design principles have shifted *AI* from a niche research topic to a major technology driver, including applications in chemistry.
88

9-
First, we outline how linear and logistic regression can be extended to so-called *single-layer neural networks*. Then, we introduce *multi-layer neural networks*, or *deep neural networks*, as a natural extension that allows for more complex and expressive models, along with efficient Python libraries for training and inference. In addition, we present some insights on the design and usage of molecular representations for machine learning.
9+
First, we outline how linear and logistic regression can be extended to so-called *single-layer neural networks*. Then, we introduce *deep neural networks* as a natural extension with multiple layers that allows for more complex and expressive models, along with efficient Python libraries for training and inference. In addition, we present fundamental concepts on the design and usage of molecular representations for machine learning.
1010

1111
| Section | Covered Examples | New Concepts and Tools |
1212
| ------- | ---------------- | ---------------------- |

‎chapters/deep_learning/single_layer_neural_networks.ipynb‎

Lines changed: 157 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,157 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"id": "7696f2f8",
6+
"metadata": {},
7+
"source": [
8+
"(sec:single_layer_neural_networks)=\n",
9+
"\n",
10+
"# Single-layer Neural Networks\n",
11+
"\n",
12+
"\n",
13+
"\n",
14+
"## Theoretical Foundations\n",
15+
"\n",
16+
"$$\n",
17+
"P(y=1|\\vec{x}) = \\frac{1}{1 + e^{-\\vec{w}^\\top \\vec{x}}} = \\sigma(\\vec{w}^\\top \\vec{x})\n",
18+
"$$\n",
19+
"\n",
20+
"$$\n",
21+
"\\hat{y}_i \n",
22+
"= \n",
23+
"\\sum_j^n a_j \\sigma(\\sum_k^d w_{kj} (\\vec{x}_i)_k)\n",
24+
"= \n",
25+
"\\vec{a}^\\top \\sigma(\\bm{W}^\\top \\vec{x}_i)\n",
26+
"$$\n",
27+
"\n",
28+
"$$\n",
29+
"\\mathcal{L} = \\sum_i^N \\left(\\hat{y}_i - y_i \\right)^2\n",
30+
"$$\n",
31+
"\n",
32+
"$$\n",
33+
"\\frac{\\partial \\mathcal{L}}{\\partial \\vec{a}} = \\sum_i^N (\\hat{y}_i - y_i) \\sigma(\\bm{W}^\\top \\vec{x}_i) \n",
34+
"$$\n",
35+
"\n",
36+
"$$\n",
37+
"\\frac{\\partial \\mathcal{L}}{\\partial \\bm{W}} = \\sum_i^N (\\hat{y}_i - y_i) \\left(\\vec{a} \\odot \\sigma'(\\bm{W}^\\top \\vec{x}_i) \\vec{x}_i^\\top \\right)^\\top\n",
38+
"$$\n",
39+
"\n",
40+
"\n",
41+
"\n",
42+
"\n",
43+
"\n",
44+
"\n",
45+
"\n",
46+
"## Implementation\n"
47+
]
48+
},
49+
{
50+
"cell_type": "code",
51+
"execution_count": 2,
52+
"id": "b4b028a4",
53+
"metadata": {},
54+
"outputs": [],
55+
"source": [
56+
"import numpy as np"
57+
]
58+
},
59+
{
60+
"cell_type": "code",
61+
"execution_count": 1,
62+
"id": "b6d85b91",
63+
"metadata": {},
64+
"outputs": [],
65+
"source": [
66+
"class Sigmoid:\n",
67+
" def __call__(self, x):\n",
68+
" return 1 / (1 + np.exp(-x))\n",
69+
"\n",
70+
" def gradient(self, x):\n",
71+
" return self(x) * (1 - self(x))"
72+
]
73+
},
74+
{
75+
"cell_type": "code",
76+
"execution_count": null,
77+
"id": "05737ca0",
78+
"metadata": {},
79+
"outputs": [],
80+
"source": [
81+
"class SLP:\n",
82+
" def __init__(self, dim=2, hidden_size=2, activation='Sigmoid', n_epochs=100, alpha=0.1, batch_size=5):\n",
83+
" self.weights = np.random.randn(dim + 1, hidden_size)\n",
84+
" self.linear_weights = np.random.randn(hidden_size)\n",
85+
" if activation == \"Sigmoid\":\n",
86+
" self.activation = Sigmoid()\n",
87+
" else:\n",
88+
" raise NotImplementedError(f\"Activation function not implemented.\")\n",
89+
" self.epochs = n_epochs\n",
90+
" self.alpha = alpha\n",
91+
" self.batch_size = batch_size\n",
92+
" \n",
93+
" def feedforward(self, x): \n",
94+
" z = np.dot(self.weights.T, x) + self.bias\n",
95+
" return np.dot(self.linear_weights.T, self.activation(z))"
96+
]
97+
},
98+
{
99+
"cell_type": "code",
100+
"execution_count": null,
101+
"id": "a348a4e8",
102+
"metadata": {},
103+
"outputs": [],
104+
"source": [
105+
" def train(self, X, y): \n",
106+
" N = X.shape[0]\n",
107+
" for e in range(self.epochs):\n",
108+
" # Iterate over batches\n",
109+
" for i in range(0, N, self.batch_size):\n",
110+
"\n",
111+
" # Define batch\n",
112+
" X_batch = X[i:i + self.batch_size]\n",
113+
" y_batch = y[i:i + self.batch_size]\n",
114+
"\n",
115+
" # Initialize gradients\n",
116+
" grad_w = np.zeros_like(self.weights)\n",
117+
" grad_lw = np.zeros_like(self.linear_weights)\n",
118+
"\n",
119+
" # Accumulate gradients over the batch\n",
120+
" for xi, yi in zip(X_batch, y_batch):\n",
121+
" \n",
122+
" zi = np.dot(self.weights.T, xi)\n",
123+
" d_inner = self.linear_weights * self.activation.gradient(zi)\n",
124+
" residue = self.feedforward(xi) - yi\n",
125+
"\n",
126+
" # Compute gradients\n",
127+
" grad_w += residue * np.outer(d_inner, xi).T\n",
128+
" grad_lw += residue * self.activation(zi)\n",
129+
"\n",
130+
" # Update parameters after each batch\n",
131+
" self.weights -= self.alpha / self.batch_size * grad_w\n",
132+
" self.linear_weights -= self.alpha / self.batch_size * grad_lw"
133+
]
134+
}
135+
],
136+
"metadata": {
137+
"kernelspec": {
138+
"display_name": ".venv",
139+
"language": "python",
140+
"name": "python3"
141+
},
142+
"language_info": {
143+
"codemirror_mode": {
144+
"name": "ipython",
145+
"version": 3
146+
},
147+
"file_extension": ".py",
148+
"mimetype": "text/x-python",
149+
"name": "python",
150+
"nbconvert_exporter": "python",
151+
"pygments_lexer": "ipython3",
152+
"version": "3.12.10"
153+
}
154+
},
155+
"nbformat": 4,
156+
"nbformat_minor": 5
157+
}

0 commit comments

Comments
 (0)