Skip to content

Commit c6dd01e

Browse files
goodways-jpgoodways-jp
committed
post new
1 parent 2f72424 commit c6dd01e

1 file changed

Lines changed: 248 additions & 0 deletions

File tree

‎_posts/2025-08-21-mastering-oracle-goldengate-obey-files.en.md‎

Lines changed: 248 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,248 @@
1+
---
2+
layout: post
3+
title: "Mastering Oracle GoldenGate OBEY Files: Building Modular and Maintainable Data Synchronization Configurations"
4+
excerpt: "This article tackles a core pain point in OGG management: monolithic and unmaintainable parameter files. We will delve into OBEY files, a powerful feature for modularizing configurations. By the end, you'll have a professional methodology for refactoring chaotic setups into clean, reusable, and maintainable configurations."
5+
date: 2025-08-21 14:02:00 +0900
6+
categories: [Oracle GoldenGate, Configuration Management, Best Practices]
7+
tags: [Oracle GoldenGate, OGG, OBEY File, Parameter File, .prm, Modular Configuration, Configuration Management, OGG Best Practices, Reusability, DRY Principle, Extract, Replicat, GGSCI, dirprm]
8+
author: Shane
9+
image: /assets/images/posts/oracle_execution_plan_overview.svg
10+
---
11+
12+
Late one night, you might receive an urgent production change request: add 10 new business tables to each of 5 running Extract processes. You take a deep breath, open the server terminal, and run `EDIT PARAMS` for each one. You carefully find the end of the `TABLE` statements in each parameter file, which are already hundreds of lines long, and then precisely copy and paste 10 lines of `TABLE schema.table;` configuration. Throughout the process, you must be fully concentrated, fearing a typo, a missing semicolon, or making a change on the wrong process, which could bring down the entire data synchronization link.
13+
14+
This scenario is a microcosm of the daily work of countless OGG administrators. As the number of tables and the complexity of business logic grow, our once-concise `.prm` parameter files inevitably evolve into a "monolithic application" that is difficult to maintain and fraught with high risk.
15+
16+
This article will solve this core pain point. We will delve into an elegant and powerful feature in Oracle GoldenGate—the `OBEY` file. After reading this article, you will master a professional methodology for refactoring chaotic configurations into a modular, reusable, and easily maintainable system, freeing you from the anxiety and inefficiency of the scenarios described above.
17+
18+
### The Root of Chaos: The Drawbacks of Monolithic Parameter Files
19+
20+
Before diving into the solution, we must clearly understand the problem. An unoptimized, massive `.prm` file typically has several fatal flaws:
21+
22+
* **Extremely Poor Readability**: Database connection information, DDL handling options, performance parameters, error handling logic, and hundreds of `TABLE` or `MAP` statements are all jumbled together, making it like reading an ancient script for newcomers.
23+
* **High Maintenance Costs**: As in the opening scenario, a simple request like "add a table" or "modify the global error handling strategy" turns into a nightmare of making repetitive changes across multiple files. This violates the most fundamental principle of software engineering: **DRY (Don't Repeat Yourself)**.
24+
* **Prone to Errors**: Manually synchronizing changes across multiple files is a classic "fat finger error" zone. Configuration inconsistencies are often the root cause of "mysterious" failures in a sync link.
25+
* **Zero Reusability**: When you need to create a new Replicat process that is highly similar to an existing one, you have no choice but to copy the entire file and then carefully modify it. The configuration cannot be reused as an independent "component."
26+
27+
These problems might be tolerable in a small-scale environment with a limited number of tables. But in a complex, enterprise-level environment, they are potential "time bombs."
28+
29+
### Getting Started with OBEY Files: Basic Syntax and Principles
30+
31+
The `OBEY` file is not a sophisticated technology; it is more of an embodiment of the "convention over configuration" design philosophy within OGG.
32+
33+
**Definition**: An `OBEY` file (usually with an `.oby` suffix, though this is not mandatory) is a plain text file containing OGG parameters. Its sole mission is to be called by a main parameter file (`.prm`) via the `OBEY` command.
34+
35+
**How It Works**: When an OGG process (like Extract or Replicat) starts and parses its `.prm` file, as soon as it encounters an `OBEY` command, it pauses parsing the current file. It then opens and fully executes all parameters within the called `OBEY` file. Once the `OBEY` file is finished, the process returns to the main file and continues parsing from the line following the `OBEY` command. This process is analogous to `#include` in C or `source` in a Shell script.
36+
37+
**Syntax**:
38+
```
39+
OBEY <file_path/file_name>
40+
```
41+
* **File Path**: You can use an absolute path, but the best practice is to use a relative path. OGG uses its startup directory as the base. By default, all our parameter files and `OBEY` files should be stored in the `dirprm` subdirectory of the OGG installation directory. This makes the configuration highly portable.
42+
43+
**A Simple Example**:
44+
Let's say we want all Replicat processes to use a unified set of database session settings.
45+
46+
1. In the `dirprm` directory, create a file named `common_settings.oby`:
47+
```
48+
-- common_settings.oby
49+
-- This file contains standard settings for all Replicat processes.
50+
DBOPTIONS SUPPRESSTRIGGERS
51+
DBOPTIONS DEFERREFCONST
52+
```
53+
54+
2. Call it from your Replicat parameter file, `rep1.prm`:
55+
56+
```
57+
-- rep1.prm
58+
REPLICAT rep1
59+
USERIDALIAS ogg_tgt_alias DOMAIN OracleGoldenGate
60+
61+
-- Include common settings
62+
OBEY common_settings.oby
63+
64+
-- Specific MAP statements for this process
65+
MAP src.customers, TARGET tgt.customers;
66+
MAP src.orders, TARGET tgt.orders;
67+
```
68+
69+
Just like that, when the `rep1` process starts, it will automatically apply the `DBOPTIONS` parameters without you having to write them repeatedly in the main file. If you need to add a new common parameter for all processes in the future, you only need to modify the `common_settings.oby` file.
70+
71+
### OBEY Files in Action: Four Classic Application Scenarios
72+
73+
The theory is simple, but the power of `OBEY` files is demonstrated in practical application scenarios. Here are four of the most widely used patterns in enterprise environments.
74+
75+
#### Scenario 1: Centralized Table List Management (`TABLE`/`MAP`)
76+
77+
This is the most basic yet most effective use of `OBEY` files.
78+
79+
* **Pain Point**: Multiple Extract processes (e.g., a real-time Extract and a batch Extract for a downstream data warehouse) need to capture the same set of core business tables.
80+
* **Solution**:
81+
1. Create a `core_tables.oby` file specifically to store the list of these tables.
82+
```
83+
-- core_tables.oby
84+
-- List of core business tables for extraction.
85+
TABLE fin.gl_ledgers;
86+
TABLE fin.gl_je_headers;
87+
TABLE fin.gl_je_lines;
88+
TABLE ar.ra_customer_trx_all;
89+
TABLE ar.ra_customer_trx_lines_all;
90+
```
91+
2. In your real-time Extract (`extfin.prm`) and batch Extract (`extdw.prm`), replace the lengthy list with a single `OBEY` line.
92+
93+
**Before (`extfin.prm`)**:
94+
```
95+
EXTRACT extfin
96+
USERIDALIAS ogg_src_alias DOMAIN OracleGoldenGate
97+
EXTTRAIL ./dirdat/fn
98+
-- Long list of tables
99+
TABLE fin.gl_ledgers;
100+
TABLE fin.gl_je_headers;
101+
TABLE fin.gl_je_lines;
102+
TABLE ar.ra_customer_trx_all;
103+
TABLE ar.ra_customer_trx_lines_all;
104+
-- ... other parameters
105+
```
106+
**After (`extfin.prm`)**:
107+
```
108+
EXTRACT extfin
109+
USERIDALIAS ogg_src_alias DOMAIN OracleGoldenGate
110+
EXTTRAIL ./dirdat/fn
111+
112+
-- Include the list of core finance tables
113+
OBEY core_tables.oby
114+
115+
-- ... other parameters
116+
```
117+
* **Benefit**: When the business needs to add a new core table (e.g., `fin.ap_invoices_all`), you only need to add one line to the `core_tables.oby` file. All processes that depend on this file will automatically pick up the change after a restart, significantly reducing the cost and risk of changes.
118+
119+
#### Scenario 2: Creating a Standardized Configuration Library
120+
121+
An enterprise-level OGG environment must have uniform standards. `OBEY` files are the perfect tool for enforcing them.
122+
123+
* **Pain Point**: How can you ensure all Replicat processes adhere to the company's standard error handling, DDL synchronization, and conflict resolution strategies?
124+
* **Solution**:
125+
* Create a "Standard Replicat Configuration Library" file, for example, `std_replicat_config.oby`.
126+
127+
```
128+
-- std_replicat_config.oby
129+
-- Standard configuration library for all Replicat processes.
130+
-- Enforces company-wide policies.
131+
132+
-- DDL Handling
133+
DDL INCLUDE MAPPED
134+
135+
-- Error Handling: Log errors for later analysis, but do not abend the process.
136+
REPERROR (DEFAULT, DISCARD)
137+
138+
-- Collision Handling: Overwrite if a record exists on insert.
139+
HANDLECOLLISIONS
140+
```
141+
* Require all new Replicat parameter files to include this `OBEY` file at the beginning.
142+
* **Benefit**: This approach transforms configuration best practices from "a document" into "a piece of executable code," enforcing consistency and stability across all sync processes. Auditing and handovers also become incredibly simple.
143+
144+
#### Scenario 3: Modularizing Complex Logic (e.g., `COLMAP`)
145+
146+
For heterogeneous replication or scenarios requiring data cleansing and transformation, the `COLMAP` clause within the `MAP` statement can become extremely bloated.
147+
148+
* **Pain Point**: A `COLMAP` with dozens of field mappings and transformation functions can make the main parameter file's readability plummet.
149+
* **Solution**:
150+
1. Separate the complex `COLMAP` logic into an independent `OBEY` file. Note that `OBEY` can be used inside a `MAP` statement.
151+
152+
**Before (`repsales.prm`)**:
153+
```
154+
REPLICAT repsales
155+
...
156+
MAP sales.orders, TARGET bi.f_orders,
157+
COLMAP (USEDEFAULTS,
158+
order_id = order_id,
159+
order_date = @DATE('YYYY-MM-DD', 'YYYY/MM/DD', order_date),
160+
customer_id = customer_id,
161+
order_total = order_value * 1.1, -- Add tax
162+
order_status = @CASE(status, 'P', 'Pending', 'S', 'Shipped', 'C', 'Cancelled', 'Unknown'),
163+
-- ... dozens more lines ...
164+
source_system = 'OLTP'
165+
);
166+
```
167+
168+
**After**:
169+
First, create the `map_orders_colmap.oby` file:
170+
```
171+
-- map_orders_colmap.oby
172+
-- Column mapping logic for the sales.orders table.
173+
COLMAP (USEDEFAULTS,
174+
order_id = order_id,
175+
order_date = @DATE('YYYY-MM-DD', 'YYYY/MM/DD', order_date),
176+
customer_id = customer_id,
177+
order_total = order_value * 1.1, -- Add tax
178+
order_status = @CASE(status, 'P', 'Pending', 'S', 'Shipped', 'C', 'Cancelled', 'Unknown'),
179+
-- ... dozens more lines ...
180+
source_system = 'OLTP'
181+
)
182+
```
183+
Then, simplify the main file `repsales.prm`:
184+
```
185+
REPLICAT repsales
186+
...
187+
MAP sales.orders, TARGET bi.f_orders,
188+
OBEY map_orders_colmap.oby;
189+
```
190+
* **Benefit**: Separation of primary and secondary logic. The main file clearly defines the "where from, where to" mapping relationship, while the specific transformation details are encapsulated in a separate module. Maintainers can quickly locate and modify the part they care about.
191+
192+
#### Scenario 4: Advanced Application - Nested OBEY and Environment Separation
193+
194+
When you have multiple environments like development, testing, and production, nested `OBEY` usage can achieve maximum configuration reuse.
195+
196+
* **Pain Point**: How can you use the same parameter file template to manage environment-specific configurations (like database connection info)?
197+
* **Solution**:
198+
1. **Define Common Configuration**: Create `common_extract_config.oby` containing parameters common to all environments (e.g., `EXTTRAIL` options).
199+
2. **Define Table List**: Create `app_tables.oby` containing the tables to be synchronized.
200+
3. **Define Environment-Specific Configuration**: Create a file for each environment.
201+
* `prod.oby`: `USERIDALIAS ogg_prod_alias DOMAIN OracleGoldenGate`
202+
* `dev.oby`: `USERIDALIAS ogg_dev_alias DOMAIN OracleGoldenGate`
203+
4. **Assemble the Main Parameter File**: The main file `extapp.prm` is only responsible for "assembly."
204+
205+
```
206+
-- extapp.prm - Main parameter file for Application Extract
207+
EXTRACT extapp
208+
209+
-- Include environment-specific settings (e.g., DB connection)
210+
OBEY dev.oby -- In DEV environment. Change to prod.oby for PROD.
211+
212+
-- Include common configurations
213+
OBEY common_extract_config.oby
214+
215+
-- Include the list of tables to be extracted
216+
OBEY app_tables.oby
217+
```
218+
* **Benefit**: This achieves the ultimate form of "Configuration as Code." 95% of the configuration (common settings and table lists) is identical across all environments, with only the environment-specific `dev.oby` or `prod.oby` being different. When deploying to a new environment, you just need to switch one `OBEY` call, greatly improving deployment efficiency and security.
219+
220+
### Best Practices and Precautions
221+
222+
To maximize the power of `OBEY` files and avoid introducing new chaos, follow these best practices:
223+
224+
1. **Establish a Clear Directory Structure**: Don't just throw all `.oby` files into the `dirprm` root directory. It's recommended to create subdirectories, for example:
225+
* `dirprm/oby/tables/`
226+
* `dirprm/oby/configs/`
227+
* `dirprm/oby/maps/`
228+
2. **Adopt a Naming Convention**: Filenames should be self-explanatory. `hr_tables.oby` is far better than `t1.oby`.
229+
3. **Embrace Version Control**: It is **essential** to put your entire `dirprm` directory under a version control system like Git. This gives you change tracking, code review, and quick rollback capabilities, which are the cornerstones of professional operations.
230+
4. **Add Thorough Comments**: In the main file, add a comment next to each `OBEY` command explaining its purpose. The `OBEY` file itself should also have a header comment.
231+
5. **Beware of Two Pitfalls**:
232+
* **Circular Dependencies**: File A `OBEY`s File B, and File B `OBEY`s File A. OGG will detect this and throw an error, but it should be avoided in the design phase.
233+
* **Parameter Overriding**: Parameters are parsed sequentially. If you define `HANDLECOLLISIONS` in an `OBEY` file and then define it again in the main file, the definition in the main file will override the one from the `OBEY` file.
234+
235+
### Conclusion
236+
237+
The `OBEY` file is not an optional "advanced trick" but a core practice of professional OGG configuration management. By breaking down "monolithic" parameter files into modular `OBEY` files, we can achieve a qualitative improvement:
238+
239+
| Core Value | Concrete Manifestation |
240+
| :------------------ | :---------------------------------------------------------- |
241+
| **Modularity** | Logic is separated; configuration units are managed independently. |
242+
| **Reusability** | "Write once, use everywhere," avoiding repetitive work. |
243+
| **Maintainability** | Change points are centralized, reducing complexity and risk. |
244+
| **Standardization** | Enforce enterprise-wide standards via a configuration library. |
245+
246+
The most important principle is: **Treat your OGG configuration as code**.
247+
248+
Starting today, review the largest and most complex `.prm` files in your environment. Try to extract the most obvious reusable part—like the table list—into an `OBEY` file. By taking this first step, you will embark on the path to more efficient and reliable Oracle GoldenGate management.

0 commit comments

Comments
 (0)