Added environment variable support for indices and environment

README.md

@@ -52,27 +52,29 @@ Our integration uses the [Splunk HEC](https://dev.splunk.com/enterprise/docs/dat
 
 Users will need to configure the HEC to accept data (enabled) and also create a new token. Steps are below.
 
-#### Create index jfrog_splunk
+#### Create index for logs (default: jfrog_splunk)
 
 ```text
 1. Open Splunk web console as administrator
 2. Click on "Settings" in dropdown select "Indexes"
 3. Click on "New Index"
-4. Enter Index name as jfrog_splunk
+4. Enter Index name as jfrog_splunk (or your custom name)
 5. Click "Save"
 ```
 
-#### Create index jfrog_splunk_metrics
+#### Create index for metrics (default: jfrog_splunk_metrics)
 
 ```text
 1. Open Splunk web console as administrator
 2. Click on "Settings" in dropdown select "Indexes"
 3. Click on "New Index"
-4. Enter Index name as jfrog_splunk_metrics
+4. Enter Index name as jfrog_splunk_metrics (or your custom name)
 5. Select Index Data Type as Metrics
 6. Click "Save"
 ```
 
+**Note:** You can customize the index names by setting the `SPLUNK_LOGS_INDEX` and `SPLUNK_METRICS_INDEX` environment variables in your docker.env file.
+
 #### Configure new HEC token to receive Logs
 
 ```text
@@ -83,7 +85,7 @@ Users will need to configure the HEC to accept data (enabled) and also create a
 5. Enter a "Name" in the textbox
 6. (Optional) Enter a "Description" in the textbox
 7. Click on the green "Next" button
-8. Add "jfrog_splunk" index to store the JFrog platform log data into.
+8. Add "jfrog_splunk" (or your custom logs index name) to store the JFrog platform log data into.
 9. Click on the green "Review" button
 10. If good, Click on the green "Done" button
 11. Save the generated token value
@@ -99,7 +101,7 @@ Users will need to configure the HEC to accept data (enabled) and also create a
 5. Enter a "Name" in the textbox
 6. (Optional) Enter a "Description" in the textbox
 7. Click on the green "Next" button
-8. Add "jfrog_splunk_metrics" index to store the JFrog platform metrics data into.
+8. Add "jfrog_splunk_metrics" (or your custom metrics index name) to store the JFrog platform metrics data into.
 9. Click on the green "Review" button
 10. If good, Click on the green "Done" button
 11. Save the generated token value
@@ -176,13 +178,16 @@ We rely heavily on environment variables so that the correct log files are strea
 * **SPLUNK_HEC_PORT**: Splunk HEC configured port
 * **SPLUNK_HEC_TOKEN**: Splunk HEC Token for sending logs to Splunk
 * **SPLUNK_METRICS_HEC_TOKEN**: Splunk HEC Token for sending metrics to Splunk
+* **SPLUNK_LOGS_INDEX**: Splunk index name for storing logs (default: jfrog_splunk)
+* **SPLUNK_METRICS_INDEX**: Splunk index name for storing metrics (default: jfrog_splunk_metrics)
 * **SPLUNK_INSECURE_SSL**: false for test environments only or if http scheme
 * **SPLUNK_VERIFY_SSL**: false for disabling ssl validation (useful for proxy forwarding or bypassing ssl certificate validation)
 * **SPLUNK_COMPRESS_DATA**: true for compressing logs and metrics json payloads on outbound to Splunk
 * **JPD_URL**: Artifactory JPD URL of the format `http://<ip_address>`
 * **JPD_ADMIN_USERNAME**: Artifactory username for authentication
 * **JFROG_ADMIN_TOKEN**: Artifactory [Access Token](https://jfrog.com/help/r/how-to-generate-an-access-token-video/artifactory-creating-access-tokens-in-artifactory) for authentication
 * **COMMON_JPD**: This flag should be set as true only for non-kubernetes installations or installations where JPD base URL is same to access both Artifactory and Xray (ex: https://sample_base_url/artifactory or https://sample_base_url/xray)
+* **LOG_ENV**: Optional environment tag for categorizing logs and metrics (e.g., `staging`, `production`, `dev`). This tag will be added to all logs and metrics sent to Splunk as `env:<value>`
 
 Apply the .env files and then run the fluentd wrapper with one argument pointed to the `fluent.conf.*` file configured.
 
@@ -234,6 +239,7 @@ For Splunk as the observability platform, execute these commands to setup the do
    **JPD_ADMIN_USERNAME**: Artifactory username for authentication
    **JFROG_ADMIN_TOKEN**: Artifactory [Access Token](https://jfrog.com/help/r/how-to-generate-an-access-token-video/artifactory-creating-access-tokens-in-artifactory) for authentication
    **COMMON_JPD**: This flag should be set as true only for non-kubernetes installations or installations where JPD base URL is same to access both Artifactory and Xray (ex: https://sample_base_url/artifactory or https://sample_base_url/xray)
+   **LOG_ENV**: Optional environment tag for categorizing logs and metrics (e.g., `staging`, `production`, `dev`). This tag will be added to all logs and metrics sent to Splunk as `env:<value>`
 3. Execute
 
    ```bash
@@ -325,6 +331,7 @@ export MASTER_KEY=$(openssl rand -hex 32)
    * **JPD_URL**: Artifactory JPD URL of the format `http://<ip_address>`
    * **JPD_ADMIN_USERNAME**: Artifactory username for authentication
    * **COMMON_JPD**: This flag should be set as true only for non-kubernetes installations or installations where JPD base URL is same to access both Artifactory and Xray (ex: https://sample_base_url/artifactory or https://sample_base_url/xray)
+   * **LOG_ENV**: Optional environment tag for categorizing logs and metrics (e.g., `staging`, `production`, `dev`). This tag will be added to all logs and metrics sent to Splunk as `env:<value>`
 
    Apply the .env files using the helm command below
 
@@ -341,18 +348,21 @@ export MASTER_KEY=$(openssl rand -hex 32)
    ```bash
    helm upgrade --install artifactory jfrog/artifactory \
           --set artifactory.joinKey=$JOIN_KEY \
-          --set databaseUpgradeReady=true --set postgresql.postgresqlPassword=$POSTGRES_PASSWORD --set nginx.service.ssloffload=true \
+          --set databaseUpgradeReady=true --set postgresql.auth.password=$POSTGRES_PASSWORD --set nginx.service.ssloffload=true \
           --set splunk.host=$SPLUNK_HEC_HOST \
           --set splunk.port=$SPLUNK_HEC_PORT \
           --set splunk.logs_token=$SPLUNK_HEC_TOKEN \
           --set splunk.metrics_token=$SPLUNK_METRICS_HEC_TOKEN \
+          --set splunk.logs_index=$SPLUNK_LOGS_INDEX \
+          --set splunk.metrics_index=$SPLUNK_METRICS_INDEX \
           --set splunk.compress_data=$SPLUNK_COMPRESS_DATA \
           --set splunk.com_protocol=$SPLUNK_COM_PROTOCOL \
           --set splunk.insecure_ssl=$SPLUNK_INSECURE_SSL \
           --set splunk.verify_ssl=$SPLUNK_VERIFY_SSL \
           --set jfrog.observability.jpd_url=$JPD_URL \
           --set jfrog.observability.username=$JPD_ADMIN_USERNAME \
           --set jfrog.observability.common_jpd=$COMMON_JPD \
+          --set jfrog.observability.log_env=$LOG_ENV \
           -f helm/artifactory-values.yaml \
           -n $INST_NAMESPACE --create-namespace
    ```
@@ -399,6 +409,7 @@ export MASTER_KEY=$(openssl rand -hex 32)
    * **JPD_URL**: Artifactory JPD URL of the format `http://<ip_address>`
    * **JPD_ADMIN_USERNAME**: Artifactory username for authentication
    * **COMMON_JPD**: This flag should be set as true only for non-kubernetes installations or installations where JPD base URL is same to access both Artifactory and Xray (ex: https://sample_base_url/artifactory or https://sample_base_url/xray)
+   * **LOG_ENV**: Optional environment tag for categorizing logs and metrics (e.g., `staging`, `production`, `dev`). This tag will be added to all logs and metrics sent to Splunk as `env:<value>`
 
    Apply the .env files and then run the helm command below
 
@@ -415,18 +426,21 @@ export MASTER_KEY=$(openssl rand -hex 32)
    ```bash
    helm upgrade --install artifactory-ha jfrog/artifactory-ha \
         --set artifactory.joinKey=$JOIN_KEY \
-        --set databaseUpgradeReady=true --set postgresql.postgresqlPassword=$POSTGRES_PASSWORD --set nginx.service.ssloffload=true \
+        --set databaseUpgradeReady=true --set postgresql.auth.password=$POSTGRES_PASSWORD --set nginx.service.ssloffload=true \
         --set splunk.host=$SPLUNK_HEC_HOST \
         --set splunk.port=$SPLUNK_HEC_PORT \
         --set splunk.logs_token=$SPLUNK_HEC_TOKEN \
         --set splunk.metrics_token=$SPLUNK_METRICS_HEC_TOKEN \
+        --set splunk.logs_index=$SPLUNK_LOGS_INDEX \
+        --set splunk.metrics_index=$SPLUNK_METRICS_INDEX \
         --set splunk.com_protocol=$SPLUNK_COM_PROTOCOL \
         --set splunk.insecure_ssl=$SPLUNK_INSECURE_SSL \
         --set splunk.verify_ssl=$SPLUNK_VERIFY_SSL \
         --set splunk.compress_data=$SPLUNK_COMPRESS_DATA \
         --set jfrog.observability.jpd_url=$JPD_URL \
         --set jfrog.observability.username=$JPD_ADMIN_USERNAME \
         --set jfrog.observability.common_jpd=$COMMON_JPD \
+        --set jfrog.observability.log_env=$LOG_ENV \
         -f helm/artifactory-ha-values.yaml \
         -n $INST_NAMESPACE --create-namespace
    ```
@@ -450,13 +464,16 @@ For Xray installation, download the .env file from [here](https://raw.githubuser
 * **SPLUNK_HEC_PORT**: Splunk HEC configured port
 * **SPLUNK_HEC_TOKEN**: Splunk HEC Token for sending logs to Splunk
 * **SPLUNK_METRICS_HEC_TOKEN**: Splunk HEC Token for sending metrics to Splunk
+* **SPLUNK_LOGS_INDEX**: Splunk index name for storing logs (default: jfrog_splunk)
+* **SPLUNK_METRICS_INDEX**: Splunk index name for storing metrics (default: jfrog_splunk_metrics)
 * **SPLUNK_INSECURE_SSL**: false for test environments only or if http scheme
 * **SPLUNK_VERIFY_SSL**: false for disabling ssl validation (useful for proxy forwarding or bypassing ssl certificate validation)
 * **SPLUNK_COMPRESS_DATA**: true for compressing logs and metrics json payloads on outbound to Splunk
 * **JPD_URL**: Artifactory JPD URL of the format `http://<ip_address>`
 * **JPD_ADMIN_USERNAME**: Artifactory username for authentication
 * **JFROG_ADMIN_TOKEN**: For security reasons, this value will be pulled from the secret jfrog-admin-token created in the step above
 * **COMMON_JPD**: This flag should be set as true only for non-kubernetes installations or installations where JPD base URL is same to access both Artifactory and Xray (ex: https://sample_base_url/artifactory or https://sample_base_url/xray)
+* **LOG_ENV**: Optional environment tag for categorizing logs and metrics (e.g., `staging`, `production`, `dev`). This tag will be added to all logs and metrics sent to Splunk as `env:<value>`
 
 Apply the .env files and then run the helm command below
 
@@ -480,13 +497,16 @@ helm upgrade --install xray jfrog/xray --set xray.jfrogUrl=$JPD_URL \
     --set splunk.port=$SPLUNK_HEC_PORT \
     --set splunk.logs_token=$SPLUNK_HEC_TOKEN \
     --set splunk.metrics_token=$SPLUNK_METRICS_HEC_TOKEN \
+    --set splunk.logs_index=$SPLUNK_LOGS_INDEX \
+    --set splunk.metrics_index=$SPLUNK_METRICS_INDEX \
     --set splunk.com_protocol=$SPLUNK_COM_PROTOCOL \
     --set splunk.insecure_ssl=$SPLUNK_INSECURE_SSL \
     --set splunk.verify_ssl=$SPLUNK_VERIFY_SSL \
     --set splunk.compress_data=$SPLUNK_COMPRESS_DATA \
     --set jfrog.observability.jpd_url=$JPD_URL \
     --set jfrog.observability.username=$JPD_ADMIN_USERNAME \
     --set jfrog.observability.common_jpd=$COMMON_JPD \
+    --set jfrog.observability.log_env=$LOG_ENV \
     -f helm/xray-values.yaml \
     -n $INST_NAMESPACE --create-namespace
 ```

app/jfrog-logs/README.md

@@ -4,11 +4,13 @@
 Install the app in your Splunk instance. Then restart your Splunk instance by going to _Server Controls > Restart_.
 
 ## Splunk Setup
-1. Create new Events index `jfrog_splunk` at _Settings > Indexes > New Index > Save_
-2. Create new Metrics index `jfrog_splunk_metrics` at _Settings > Indexes > New Index > Metrics > Save_
+1. Create new Events index `jfrog_splunk` (or your custom name) at _Settings > Indexes > New Index > Save_
+2. Create new Metrics index `jfrog_splunk_metrics` (or your custom name) at _Settings > Indexes > New Index > Metrics > Save_
 3. Create a new HTTP Event Collector data input for logs at _Settings > Data Inputs > HTTP Event Collector > New Token > jfrog_splunk index > Save_
 4. Create a new HTTP Event Collector data input for metrics at _Settings > Data Inputs > HTTP Event Collector > New Token > jfrog_splunk_metrics index > Save_
 
+**Note:** You can customize the index names by setting the `SPLUNK_LOGS_INDEX` and `SPLUNK_METRICS_INDEX` environment variables in your configuration.
+
 ## Setup Fluentd
 FluentD is used to send log events to Splunk. This [repo](https://github.com/jfrog/log-analytics-splunk) contains instructions on various installations options for Fluentd as a logging agent. 
 
@@ -21,6 +23,8 @@ export SPLUNK_HEC_HOST=splunk.example.com
 export SPLUNK_HEC_PORT=8088
 export SPLUNK_HEC_TOKEN=SPLUNK_HEC_TOKEN
 export SPLUNK_METRICS_HEC_TOKEN=SPLUNK_METRICS_HEC_TOKEN
+export SPLUNK_LOGS_INDEX=jfrog_splunk
+export SPLUNK_METRICS_INDEX=jfrog_splunk_metrics
 export SPLUNK_INSECURE_SSL=false
 export SPLUNK_VERIFY_SSL=true
 export SPLUNK_COMPRESS_DATA=true
@@ -36,6 +40,8 @@ export COMMON_JPD=false
 * **SPLUNK_HEC_PORT**: Splunk HEC configured port
 * **SPLUNK_HEC_TOKEN**: Splunk HEC Token for sending logs to Splunk
 * **SPLUNK_METRICS_HEC_TOKEN**: Splunk HEC Token for sending metrics to Splunk
+* **SPLUNK_LOGS_INDEX**: Splunk index name for storing logs (default: jfrog_splunk)
+* **SPLUNK_METRICS_INDEX**: Splunk index name for storing metrics (default: jfrog_splunk_metrics)
 * **SPLUNK_INSECURE_SSL**: false for test environments only or if http scheme
 * **SPLUNK_VERIFY_SSL**: false for disabling ssl validation (useful for proxy forwarding or bypassing ssl certificate validation)
 * **SPLUNK_COMPRESS_DATA**: true for compressing logs and metrics json payloads on outbound to Splunk

app/jfrog-logs/default/macros.conf

@@ -1,7 +1,7 @@
 [default_index]
-definition = index="jfrog_splunk"
+definition = index="$SPLUNK_LOGS_INDEX$"
 iseval = 0
 
 [default_metrics_index]
-definition = "index"="jfrog_splunk_metrics"
+definition = "index"="$SPLUNK_METRICS_INDEX$"
 iseval = 0

docker-build/docker.env

@@ -4,10 +4,13 @@ SPLUNK_HEC_HOST=splunk.change_me.com
 SPLUNK_HEC_PORT=8088
 SPLUNK_HEC_TOKEN=change_me
 SPLUNK_METRICS_HEC_TOKEN=change_me
+SPLUNK_LOGS_INDEX=jfrog_splunk
+SPLUNK_METRICS_INDEX=jfrog_splunk_metrics
 SPLUNK_INSECURE_SSL=change_me
 SPLUNK_VERIFY_SSL=change_me
 SPLUNK_COMPRESS_DATA=change_me
 JPD_URL=https://change_me.jfrog.io
 JPD_ADMIN_USERNAME=admin
 JFROG_ADMIN_TOKEN=change_me
 COMMON_JPD=false
+LOG_ENV=production

fluent.conf.rt

@@ -23,7 +23,7 @@
   hec_token "#{ENV['SPLUNK_METRICS_HEC_TOKEN']}"
   flush_interval 5s
   source ${tag}
-  index jfrog_splunk_metrics
+  index "#{ENV['SPLUNK_METRICS_INDEX']}"
   metric_name_key metric_name
   metric_value_key value
   insecure_ssl "#{ENV['SPLUNK_INSECURE_SSL']}"
@@ -379,6 +379,7 @@
     <record>
       hostname "#{Socket.gethostname}"
       log_source ${tag}
+      env "#{ENV['LOG_ENV']}"
     </record>
   </filter>
 
@@ -395,7 +396,7 @@
     hec_host "#{ENV['SPLUNK_HEC_HOST']}"
     hec_port "#{ENV['SPLUNK_HEC_PORT']}"
     hec_token "#{ENV['SPLUNK_HEC_TOKEN']}"
-    index jfrog_splunk
+    index "#{ENV['SPLUNK_LOGS_INDEX']}"
     format json
     sourcetype_key log_source
     use_fluentd_time false

fluent.conf.xray

@@ -23,7 +23,7 @@
   hec_token "#{ENV['SPLUNK_METRICS_HEC_TOKEN']}"
   flush_interval 5s
   source ${tag}
-  index jfrog_splunk_metrics
+  index "#{ENV['SPLUNK_METRICS_INDEX']}"
   metric_name_key metric_name
   metric_value_key value
   insecure_ssl "#{ENV['SPLUNK_INSECURE_SSL']}"
@@ -147,7 +147,7 @@
   hec_host "#{ENV['SPLUNK_HEC_HOST']}"
   hec_port "#{ENV['SPLUNK_HEC_PORT']}"
   hec_token "#{ENV['SPLUNK_HEC_TOKEN']}"
-  index jfrog_splunk
+  index "#{ENV['SPLUNK_LOGS_INDEX']}"
   format json
   sourcetype_key log_source
   use_fluentd_time false
@@ -344,6 +344,7 @@
     <record>
       hostname "#{Socket.gethostname}"
       log_source ${tag}
+      env "#{ENV['LOG_ENV']}"
     </record>
   </filter>
 
@@ -357,7 +358,7 @@
     hec_host "#{ENV['SPLUNK_HEC_HOST']}"
     hec_port "#{ENV['SPLUNK_HEC_PORT']}"
     hec_token "#{ENV['SPLUNK_HEC_TOKEN']}"
-    index jfrog_splunk
+    index "#{ENV['SPLUNK_LOGS_INDEX']}"
     format json
     sourcetype_key log_source
     use_fluentd_time false

fluentd-demo.conf

@@ -185,7 +185,7 @@
   hec_host HEC_HOST
   hec_port HEC_PORT
   hec_token HEC_TOKEN
-  index jfrog_splunk
+  index "#{ENV['SPLUNK_LOGS_INDEX']}"
   format json
   sourcetype_key log_source
   use_fluentd_time false

helm/artifactory-ha-values.yaml

@@ -47,19 +47,27 @@ artifactory:
           value: {{ .Values.splunk.metrics_token }}
         - name: SPLUNK_HEC_TOKEN
           value: {{ .Values.splunk.logs_token }}
+        - name: SPLUNK_LOGS_INDEX
+          value: {{ .Values.splunk.logs_index | default "jfrog_splunk" }}
+        - name: SPLUNK_METRICS_INDEX
+          value: {{ .Values.splunk.metrics_index | default "jfrog_splunk_metrics" }}
         - name: SPLUNK_INSECURE_SSL
           value: {{ .Values.splunk.insecure_ssl | quote}}
         - name: SPLUNK_VERIFY_SSL
           value: {{ .Values.splunk.verify_ssl | quote}}  
         - name: SPLUNK_COMPRESS_DATA
           value: {{ .Values.splunk.compress_data | quote}}
+        - name: LOG_ENV
+          value: {{ .Values.jfrog.observability.log_env | default "production" }}
         - name: FLUENTD_CONF
           value: ../../../..{{ .Values.artifactory.persistence.mountPath }}/etc/fluentd/fluentd.conf
 splunk:
   host: SPLUNK_HEC_HOST
   port: SPLUNK_HEC_PORT
   logs_token: SPLUNK_HEC_TOKEN
   metrics_token: SPLUNK_METRICS_HEC_TOKEN
+  logs_index: SPLUNK_LOGS_INDEX
+  metrics_index: SPLUNK_METRICS_INDEX
   com_protocol: SPLUNK_COM_PROTOCOL
   insecure_ssl: SPLUNK_INSECURE_SSL
   verify_ssl: SPLUNK_VERIFY_SSL
@@ -68,4 +76,5 @@ jfrog:
   observability:
     jpd_url: JPD_URL
     username: JPD_ADMIN_USERNAME
-    common_jpd: COMMON_JPD
+    common_jpd: COMMON_JPD
+    log_env: LOG_ENV

helm/artifactory-values.yaml

@@ -47,19 +47,27 @@ artifactory:
           value: {{ .Values.splunk.metrics_token }}
         - name: SPLUNK_HEC_TOKEN
           value: {{ .Values.splunk.logs_token }}
+        - name: SPLUNK_LOGS_INDEX
+          value: {{ .Values.splunk.logs_index | default "jfrog_splunk" }}
+        - name: SPLUNK_METRICS_INDEX
+          value: {{ .Values.splunk.metrics_index | default "jfrog_splunk_metrics" }}
         - name: SPLUNK_INSECURE_SSL
           value: {{ .Values.splunk.insecure_ssl | quote}}
         - name: SPLUNK_VERIFY_SSL
           value: {{ .Values.splunk.verify_ssl | quote}}  
         - name: SPLUNK_COMPRESS_DATA
           value: {{ .Values.splunk.compress_data | quote }}
+        - name: LOG_ENV
+          value: {{ .Values.jfrog.observability.log_env | default "production" }}
         - name: FLUENTD_CONF
           value: ../../../..{{ .Values.artifactory.persistence.mountPath }}/etc/fluentd/fluentd.conf
 splunk:
   host: SPLUNK_HEC_HOST
   port: SPLUNK_HEC_PORT
   logs_token: SPLUNK_HEC_TOKEN
   metrics_token: SPLUNK_METRICS_HEC_TOKEN
+  logs_index: SPLUNK_LOGS_INDEX
+  metrics_index: SPLUNK_METRICS_INDEX
   com_protocol: SPLUNK_COM_PROTOCOL
   insecure_ssl: SPLUNK_INSECURE_SSL
   verify_ssl: SPLUNK_VERIFY_SSL
@@ -68,4 +76,5 @@ jfrog:
   observability:
     jpd_url: JPD_URL
     username: JPD_ADMIN_USERNAME
-    common_jpd: COMMON_JPD
+    common_jpd: COMMON_JPD
+    log_env: LOG_ENV

helm/jfrog_helm.env

@@ -3,9 +3,12 @@ export SPLUNK_HEC_HOST=splunk.example.com
 export SPLUNK_HEC_PORT=8088
 export SPLUNK_HEC_TOKEN=SPLUNK_HEC_TOKEN
 export SPLUNK_METRICS_HEC_TOKEN=SPLUNK_METRICS_HEC_TOKEN
+export SPLUNK_LOGS_INDEX=jfrog_splunk
+export SPLUNK_METRICS_INDEX=jfrog_splunk_metrics
 export SPLUNK_INSECURE_SSL=false
 export SPLUNK_VERIFY_SSL=true
 export SPLUNK_COMPRESS_DATA=true
 export JPD_URL=http://abc.jfrog.io
 export JPD_ADMIN_USERNAME=admin
-export COMMON_JPD=false
+export COMMON_JPD=false
+export LOG_ENV=staging