Skip to content

Commit e7dad96

Browse files
ngopalak-redhatngopalak-redhat
committed
Add claude md file for using the node_utils
1 parent 653bd69 commit e7dad96

7 files changed

Lines changed: 388 additions & 160 deletions

File tree

‎test/extended/node/CLAUDE.md‎

Lines changed: 190 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,190 @@
1+
# OpenShift Node E2E Tests - Developer Guide
2+
3+
## Overview
4+
5+
This directory contains E2E tests for OpenShift node-related functionality using the Ginkgo test framework. When writing tests here, **ALWAYS use the utility functions in `node_utils.go`** instead of implementing your own.
6+
7+
## Key Utility Functions Available
8+
9+
**See `node_utils.go` for current function signatures.** The lists below describe exported (public) functions only. Lowercase helper functions are internal implementation details and not documented here.
10+
11+
### Node Selection and Filtering
12+
13+
- `GetNodesByLabel()` - Get nodes matching a label selector
14+
- `GetControlPlaneNodes()` - Get all control plane nodes (handles both master and control-plane labels)
15+
- `GetPureWorkerNodes()` - Filter out nodes that have both worker and control-plane roles (important for SNO clusters)
16+
- `GetCNVWorkerNodeName()` - Get a randomly selected CNV-enabled worker node
17+
18+
### Executing Commands on Nodes
19+
20+
- `ExecOnNodeWithChroot()` - Run command on a node using `oc debug` with `chroot /host` (most common)
21+
- `ExecOnNodeWithNsenter()` - Run command on a node using `nsenter` to access host namespaces (required for swap operations)
22+
23+
**Important**: These functions handle the `oc debug` boilerplate for you. Never manually construct `oc debug node/...` commands.
24+
25+
### Managing Kubelet Configuration
26+
27+
- `GetKubeletConfigFromNode()` - Get kubelet configuration from a node via the configz API
28+
- `CreateDropInFile()` - Create a drop-in configuration file on a node
29+
- `RemoveDropInFile()` - Remove a drop-in configuration file from a node
30+
- `LoadConfigFromFile()` - Read kubelet configuration from a YAML file in testdata
31+
- `EnsureDropInDirectoryExists()` - Create drop-in directory on all worker nodes if it doesn't exist
32+
33+
### Kubelet Lifecycle Management
34+
35+
- `RestartKubeletOnNode()` - Restart kubelet service on a node (with automatic retry on transient network errors)
36+
- `WaitForNodeToBeReady()` - Wait for a node to reach Ready condition
37+
- `IsNodeInReadyState()` - Check if a node is currently in Ready condition
38+
- `CleanupDropInAndRestartKubelet()` - Remove drop-in file, restart kubelet, and wait for node Ready (cleanup pattern)
39+
40+
### CNV (OpenShift Virtualization) Operations
41+
42+
- `IsCNVInstalled()` - Check if CNV operator is installed in the cluster
43+
- `InstallCNVOperator()` - Install CNV operator (creates namespace, subscription, HyperConverged CR, labels nodes, waits for MCP)
44+
- `UninstallCNVOperator()` - Uninstall CNV operator and clean up all resources
45+
- `LabelWorkerNodesForCNV()` - Label all worker nodes with `kubevirt.io/schedulable=true`
46+
- `UnlabelWorkerNodesForCNV()` - Remove CNV scheduling labels from worker nodes
47+
48+
### MachineConfigPool Operations
49+
50+
- `WaitForMCP()` - Wait for a MachineConfigPool to finish updating (returns error if degraded)
51+
- `GetWorkerGeneratedKubeletMC()` - Get the highest numbered `worker-generated-kubelet` MachineConfig
52+
53+
## Test Structure Best Practices
54+
55+
**Note:** Code examples below are illustrative. Check `node_utils.go` for current function signatures.
56+
57+
### Standard Test Pattern
58+
59+
```go
60+
var _ = g.Describe("[sig-node][Feature:MyFeature] Description", func() {
61+
defer g.GinkgoRecover()
62+
63+
var oc = exutil.NewCLI("test-name")
64+
65+
g.BeforeAll(func(ctx context.Context) {
66+
// Setup that applies to all tests in this Describe block
67+
})
68+
69+
g.AfterAll(func(ctx context.Context) {
70+
// Cleanup after all tests
71+
})
72+
73+
g.It("should do something", func(ctx context.Context) {
74+
// Test implementation using context.Context
75+
nodes, err := GetNodesByLabel(ctx, oc, "node-role.kubernetes.io/worker")
76+
o.Expect(err).NotTo(o.HaveOccurred())
77+
})
78+
})
79+
```
80+
81+
### Always Use Context
82+
83+
Always pass `ctx` when the helper or test function signature requires it. Most utility functions in `node_utils.go` accept `context.Context` (check the function signature). Ginkgo test functions should always use the `ctx` parameter:
84+
85+
```go
86+
// Good - test function receives and uses ctx
87+
g.It("test name", func(ctx context.Context) {
88+
nodes, err := GetNodesByLabel(ctx, oc, "label") // Helper needs ctx
89+
})
90+
91+
// Bad - missing context parameter
92+
g.It("test name", func() {
93+
// Cannot call context-aware helpers
94+
})
95+
```
96+
97+
**Note:** Some helpers like `ExecOnNodeWithChroot()` and `CreateDropInFile()` do not require `ctx`. Check the function signature in `node_utils.go`.
98+
99+
### Common Patterns
100+
101+
**Note:** Examples below are illustrative. Check `node_utils.go` for current function signatures.
102+
103+
**Finding a worker node:**
104+
```go
105+
allWorkerNodes, err := GetNodesByLabel(ctx, oc, "node-role.kubernetes.io/worker")
106+
o.Expect(err).NotTo(o.HaveOccurred())
107+
o.Expect(len(allWorkerNodes)).Should(o.BeNumerically(">", 0))
108+
109+
// Filter out nodes that are also control plane (SNO handling)
110+
workerNodes := GetPureWorkerNodes(allWorkerNodes)
111+
o.Expect(len(workerNodes)).Should(o.BeNumerically(">", 0), "expected at least one pure worker node")
112+
nodeName := workerNodes[0].Name
113+
```
114+
115+
**Safely modifying kubelet config:**
116+
```go
117+
// In test setup
118+
dropInPath := "/etc/kubelet.conf.d/99-my-test.conf"
119+
configContent := LoadConfigFromFile(exutil.FixturePath("testdata", "node", "my-config.yaml"))
120+
err := CreateDropInFile(oc, nodeName, dropInPath, configContent)
121+
o.Expect(err).NotTo(o.HaveOccurred())
122+
123+
err = RestartKubeletOnNode(ctx, oc, nodeName)
124+
o.Expect(err).NotTo(o.HaveOccurred())
125+
126+
WaitForNodeToBeReady(ctx, oc, nodeName)
127+
128+
// In cleanup (defer or AfterEach)
129+
CleanupDropInAndRestartKubelet(ctx, oc, nodeName, dropInPath)
130+
```
131+
132+
**Working with CNV:**
133+
```go
134+
var cnvInstalledByTest bool
135+
136+
g.BeforeAll(func(ctx context.Context) {
137+
if !IsCNVInstalled(ctx, oc) {
138+
err := InstallCNVOperator(ctx, oc)
139+
if err != nil {
140+
e2eskipper.Skipf("Failed to install CNV: %v", err)
141+
}
142+
cnvInstalledByTest = true
143+
}
144+
})
145+
146+
g.AfterAll(func(ctx context.Context) {
147+
if cnvInstalledByTest {
148+
UninstallCNVOperator(ctx, oc)
149+
}
150+
})
151+
```
152+
153+
## Common Mistakes to Avoid
154+
155+
1. **Don't manually construct `oc debug` commands** - use `ExecOnNodeWithChroot()` or `ExecOnNodeWithNsenter()`
156+
157+
2. **Don't forget to handle SNO clusters** - use `GetPureWorkerNodes()` to filter out nodes with dual roles
158+
159+
3. **Don't skip context propagation** - always pass `ctx` to utility functions
160+
161+
4. **Don't forget cleanup** - use `defer` or `g.AfterEach` with `CleanupDropInAndRestartKubelet()`
162+
163+
5. **Don't ignore MCP rollouts** - after MachineConfig changes, use `WaitForMCP()` to ensure stability
164+
165+
6. **Don't assume swap operations work with chroot** - use `ExecOnNodeWithNsenter()` for swap commands
166+
167+
## Constants and GVRs Available
168+
169+
The file defines commonly used constants:
170+
- `debugNamespace = "openshift-machine-config-operator"`
171+
- `cnvNamespace = "openshift-cnv"`
172+
- CNV-related resource names
173+
174+
And GVRs for dynamic client operations:
175+
- `subscriptionGVR`, `operatorGroupGVR`, `hyperConvergedGVR`, `csvGVR`, `mcpGVR`
176+
177+
## Example: Complete Test
178+
179+
See `node_swap_cnv.go` for a complete example showing:
180+
- BeforeAll/AfterAll hooks
181+
- CNV installation/cleanup
182+
- Using multiple utility functions together
183+
- Proper error handling and skip conditions
184+
- Working with drop-in files and kubelet restarts
185+
186+
## Getting Help
187+
188+
- Read the function documentation in `node_utils.go`
189+
- Look at existing tests in this directory for patterns
190+
- Check testdata files in `testdata/node/` for config examples

‎test/extended/node/claude-docs-check.sh‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
#!/bin/bash
2+
# Check if node_utils.go has functions not mentioned in CLAUDE.md
3+
4+
set -e
5+
6+
NODE_UTILS="test/extended/node/node_utils.go"
7+
CLAUDE_MD="test/extended/node/CLAUDE.md"
8+
9+
# Extract ONLY exported function names from node_utils.go (start with uppercase)
10+
# Lowercase (unexported) helpers are intentionally not documented in CLAUDE.md
11+
# Matches both standalone functions and receiver methods, including digits in names
12+
UTILS_FUNCS=$(
13+
grep -E '^[[:space:]]*func([[:space:]]+\([^)]*\))?[[:space:]]+[A-Z][A-Za-z0-9_]*[[:space:]]*\(' "$NODE_UTILS" \
14+
| sed -E 's/^[[:space:]]*func([[:space:]]+\([^)]*\))?[[:space:]]+([A-Z][A-Za-z0-9_]*)[[:space:]]*\(.*/\2/' \
15+
| sort -u
16+
)
17+
18+
# Read CLAUDE.md once for efficiency
19+
CLAUDE_CONTENT=$(cat "$CLAUDE_MD")
20+
21+
# Check each function is mentioned in CLAUDE.md (word-boundary match to avoid false positives)
22+
MISSING=()
23+
for func in $UTILS_FUNCS; do
24+
if ! echo "$CLAUDE_CONTENT" | grep -Fqw "$func"; then
25+
MISSING+=(" - $func()")
26+
fi
27+
done
28+
29+
if [ ${#MISSING[@]} -gt 0 ]; then
30+
echo "⚠️ Warning: node_utils.go functions not documented in CLAUDE.md:"
31+
printf '%s\n' "${MISSING[@]}"
32+
echo ""
33+
echo "Please update CLAUDE.md to document these utility functions."
34+
exit 1
35+
fi
36+
37+
echo "✅ All node_utils.go functions are documented in CLAUDE.md"

‎test/extended/node/node_sizing.go‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -151,7 +151,7 @@ var _ = g.Describe("[Suite:openshift/disruptive-longrunning][sig-node][Disruptiv
151151
g.DeferCleanup(cleanupMCP)
152152

153153
g.By("Waiting for custom MachineConfigPool to be ready")
154-
err = waitForMCP(ctx, mcClient, testMCPName, 5*time.Minute)
154+
err = WaitForMCP(ctx, mcClient, testMCPName, 5*time.Minute)
155155
o.Expect(err).NotTo(o.HaveOccurred(), "Custom MachineConfigPool should become ready")
156156

157157
verifyNodeSizingEnabledFile(oc, nodeName, "true")
@@ -193,7 +193,7 @@ var _ = g.Describe("[Suite:openshift/disruptive-longrunning][sig-node][Disruptiv
193193

194194
// Wait for custom MCP to be ready after cleanup
195195
g.By("Waiting for custom MCP to be ready after KubeletConfig deletion")
196-
waitErr := waitForMCP(cleanupCtx, mcClient, testMCPName, 5*time.Minute)
196+
waitErr := WaitForMCP(cleanupCtx, mcClient, testMCPName, 5*time.Minute)
197197
if apierrors.IsNotFound(waitErr) {
198198
// MachineConfigPool already deleted, nothing to wait for
199199
} else if waitErr != nil {
@@ -229,7 +229,7 @@ var _ = g.Describe("[Suite:openshift/disruptive-longrunning][sig-node][Disruptiv
229229
}, 2*time.Minute, 10*time.Second).Should(o.BeTrue(), fmt.Sprintf("%s MCP should start updating", testMCPName))
230230

231231
g.By(fmt.Sprintf("Waiting for %s MCP to be ready with new configuration", testMCPName))
232-
err = waitForMCP(ctx, mcClient, testMCPName, 15*time.Minute)
232+
err = WaitForMCP(ctx, mcClient, testMCPName, 15*time.Minute)
233233
o.Expect(err).NotTo(o.HaveOccurred(), fmt.Sprintf("%s MCP should become ready with new configuration", testMCPName))
234234

235235
verifyNodeSizingEnabledFile(oc, nodeName, "false")

‎test/extended/node/node_swap.go‎

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -45,16 +45,16 @@ var _ = g.Describe("[Jira:Node][sig-node] Node non-cnv swap configuration", func
4545
// the kubelet will not use it for memory management, maintaining consistent behavior across the cluster.
4646
g.It("should have correct default kubelet swap settings with worker nodes failSwapOn=false, control plane nodes failSwapOn=true, and both swapBehavior=NoSwap [OCP-86394]", ote.Informing(), func(ctx context.Context) {
4747
g.By("Getting worker nodes")
48-
allWorkerNodes, err := getNodesByLabel(ctx, oc, "node-role.kubernetes.io/worker")
48+
allWorkerNodes, err := GetNodesByLabel(ctx, oc, "node-role.kubernetes.io/worker")
4949
o.Expect(err).NotTo(o.HaveOccurred())
5050
o.Expect(len(allWorkerNodes)).Should(o.BeNumerically(">", 0), "Expected at least one worker node")
5151

5252
// Filter out nodes that are also control plane (e.g., SNO)
53-
workerNodes := getPureWorkerNodes(allWorkerNodes)
53+
workerNodes := GetPureWorkerNodes(allWorkerNodes)
5454

5555
g.By("Validating kubelet configuration on each worker node")
5656
for _, node := range workerNodes {
57-
config, err := getKubeletConfigFromNode(ctx, oc, node.Name)
57+
config, err := GetKubeletConfigFromNode(ctx, oc, node.Name)
5858
o.Expect(err).NotTo(o.HaveOccurred(), "Failed to get kubelet config for worker node %s", node.Name)
5959

6060
g.By(fmt.Sprintf("Checking failSwapOn=false on worker node %s", node.Name))
@@ -74,13 +74,13 @@ var _ = g.Describe("[Jira:Node][sig-node] Node non-cnv swap configuration", func
7474

7575
if *controlPlaneTopology != configv1.ExternalTopologyMode {
7676
g.By("Getting control plane nodes")
77-
controlPlaneNodes, err := getControlPlaneNodes(ctx, oc)
77+
controlPlaneNodes, err := GetControlPlaneNodes(ctx, oc)
7878
o.Expect(err).NotTo(o.HaveOccurred())
7979
o.Expect(len(controlPlaneNodes)).Should(o.BeNumerically(">", 0), "Expected at least one control plane node")
8080

8181
g.By("Validating kubelet configuration on each control plane node")
8282
for _, node := range controlPlaneNodes {
83-
config, err := getKubeletConfigFromNode(ctx, oc, node.Name)
83+
config, err := GetKubeletConfigFromNode(ctx, oc, node.Name)
8484
o.Expect(err).NotTo(o.HaveOccurred(), "Failed to get kubelet config for control plane node %s", node.Name)
8585

8686
g.By(fmt.Sprintf("Checking failSwapOn=true on control plane node %s", node.Name))
@@ -113,7 +113,7 @@ var _ = g.Describe("[Jira:Node][sig-node] Node non-cnv swap configuration", func
113113

114114
g.By("Getting initial machine config resourceVersion")
115115
// Get the initial resourceVersion of the worker machine config before creating KubeletConfig
116-
workerGeneratedKubeletMC, err := getWorkerGeneratedKubeletMC(ctx, mcClient)
116+
workerGeneratedKubeletMC, err := GetWorkerGeneratedKubeletMC(ctx, mcClient)
117117
o.Expect(err).NotTo(o.HaveOccurred(), "Failed to find worker-generated-kubelet MachineConfig")
118118
initialResourceVersion := workerGeneratedKubeletMC.ResourceVersion
119119
framework.Logf("Initial %s resourceVersion: %s", workerGeneratedKubeletMC.Name, initialResourceVersion)
@@ -183,21 +183,21 @@ var _ = g.Describe("[Jira:Node][sig-node] Node non-cnv swap configuration", func
183183
time.Sleep(5 * time.Second)
184184

185185
// Check if the machine config was created or updated (compare to initial resourceVersion captured earlier)
186-
workerMCAfter, err := getWorkerGeneratedKubeletMC(ctx, mcClient)
186+
workerMCAfter, err := GetWorkerGeneratedKubeletMC(ctx, mcClient)
187187
o.Expect(err).NotTo(o.HaveOccurred(), "Failed to find worker-generated-kubelet MachineConfig for verification")
188188
o.Expect(workerMCAfter.ResourceVersion).To(o.Equal(initialResourceVersion), "Machine config %s should not be updated when failSwapOn is rejected", workerMCAfter.Name)
189189
framework.Logf("Verified: %s was not updated (resourceVersion: %s)", workerMCAfter.Name, workerMCAfter.ResourceVersion)
190190

191191
g.By("Verifying worker nodes still have correct swap settings")
192-
allWorkerNodes, err := getNodesByLabel(ctx, oc, "node-role.kubernetes.io/worker")
192+
allWorkerNodes, err := GetNodesByLabel(ctx, oc, "node-role.kubernetes.io/worker")
193193
o.Expect(err).NotTo(o.HaveOccurred())
194194
o.Expect(len(allWorkerNodes)).Should(o.BeNumerically(">", 0), "Expected at least one worker node")
195195

196196
// Filter out nodes that are also control plane (e.g., SNO)
197-
workerNodes := getPureWorkerNodes(allWorkerNodes)
197+
workerNodes := GetPureWorkerNodes(allWorkerNodes)
198198

199199
for _, node := range workerNodes {
200-
config, err := getKubeletConfigFromNode(ctx, oc, node.Name)
200+
config, err := GetKubeletConfigFromNode(ctx, oc, node.Name)
201201
o.Expect(err).NotTo(o.HaveOccurred(), "Failed to get kubelet config for worker node %s", node.Name)
202202

203203
g.By(fmt.Sprintf("Verifying failSwapOn=false remains unchanged on worker node %s", node.Name))

0 commit comments

Comments
 (0)