December Development#184
Merged
MattPCollins merged 35 commits intomainfrom Dec 9, 2025
Merged
Conversation
Collaborator
MattPCollins
commented
Dec 8, 2025
MattPCollins
commented
- Latest CICD improvements
- Infrastructure deployment robustness
- Fabric compatible Azure Functions
- Simplified Metadata-As-Code deployments
- Control demo pipelines
- Improved Orchestration Lineage diagrams
- Simplified notebook execution logic
- New ingest data source connectors
Forward merge to keep develop in sync
* Update packages * Published functionapp with updated NuGet packages * Update .gitignore
* Overhaul PowerShell and Bicep 1-click deployment functionality * Update Bicep configurations * Update Databricks Deployment options * Update bicep deployment logic - tidy up namings and small bug fix with transform * Update gitignore for Functions App
* Adding linkedService: Ingest_LS_Jira_UserAuth * Updating linkedService: Ingest_LS_Jira_UserAuth * Updating linkedService: Ingest_LS_Jira_UserAuth * Updating linkedService: Ingest_LS_Jira_UserAuth * Updating linkedService: Ingest_LS_Jira_UserAuth * Updating linkedService: Ingest_LS_Jira_UserAuth * Updating linkedService: Ingest_LS_Jira_UserAuth * Adding JiraObject: Ingest_DS_Jira_UserAuth * Updating JiraObject: Ingest_DS_Jira_UserAuth * Adding pipeline: Ingest_PL_Jira * Updating pipeline: Ingest_PL_Jira * Updating pipeline: Ingest_PL_Jira * Updating pipeline: Ingest_PL_Jira * Updating pipeline: Ingest_PL_Jira * Updating pipeline: Ingest_PL_Jira * Update DefaultConnectionTypes.sql
* Adding linkedService: Ingest_LS_Oracle_V2_SIDAuth * Adding integrationRuntime: integrationRuntimeCumulusDataSources02 * Updating linkedService: Ingest_LS_Oracle_SIDAuth * Updating linkedService: Ingest_LS_Oracle_V2_SIDAuth * Updating linkedService: Ingest_LS_Oracle_V2_SIDAuth * Adding OracleTable: Ingest_DS_Oracle_V2_SIDAuth * Updating pipeline: Ingest_PL_Oracle * Updating pipeline: Ingest_PL_Oracle * Updating pipeline: Ingest_PL_Oracle * Update GetDatasetPayload.sql removed PSQL from the clause that adds a semicolon * Updating linkedService: Ingest_LS_Oracle_V2_SIDAuth * Updating linkedService: Ingest_LS_Oracle_SIDAuth * Updated gitignore to exclude integrationRuntime folder --------- Co-authored-by: MattPCollins <34279269+MattPCollins@users.noreply.github.com>
* Databricks Module restructure * refactor notebooks to functions * Update file references * Update Ingest to reference new python files rather than notebooks * refactor pep8 naming * refactor pep8 naming convention * refactor pep8 naming conventions * integration test changes for pep8 formatting * refactor transform python pep8 naming conventions * Add python file references to Transform notebooks * Integration tests new Transform Managed Notebooks * Add further ingest tests and utils.HelperFunctions final tests * Add unit tests for remaining CheckPayloadFunctions * Update automated testing for python unit tests * update unit test workflow * amend pytest file path * amend pytest file path * Update pytest runner for correct paths --------- Co-authored-by: MattPCollins <MattPCollins>
* Additional utilities unit tests * update automated testing for python unit tests
* Adding linkedService: Common_LS_Databricks_Cluster_MIAuth * Updating linkedService: Common_LS_Databricks_Cluster_MIAuth * Updating pipeline: Ingest_PL_Merge * Updating pipeline: Transform_PL_Managed * Updating pipeline: Transform_PL_Unmanaged * Updating pipeline: Transform_PL_Unmanaged * Updating pipeline: Transform_PL_Managed * Adding linkedService: Common_LS_Databricks_JobCluster_MIAuth * Updating linkedService: Common_LS_Databricks_JobCluster_MIAuth * Updating pipeline: Transform_PL_Managed * Updating pipeline: Transform_PL_Unmanaged * Updating pipeline: Transform_PL_Managed * Updating pipeline: Ingest_PL_Merge * Update MetadataAsCodeCommon Script for updated LS names * Deleting linkedService: Transform_LS_Databricks_Cluster_MIAuth * Deleting linkedService: Ingest_LS_Databricks_Cluster_MIAuth * Deleting linkedService: Ingest_LS_Databricks_JobCluster_MIAuth --------- Co-authored-by: MattPCollins <34279269+MattPCollins@users.noreply.github.com>
* Update Databricks CICD process * Update Databricks CICD process2 * Test DAB profile config * Test DAB profile config with script correction * Test DAB profile config with script correction * Extend DAB logic to test/prod and rename WS file refernces for databricks in ADF pipelines and metadata. * Databricks complete deploy in PS
* Update adf-cicd.yml adding additional parameters * Updating linkedService: Ingest_LS_Oracle_SIDAuth * Updating pipeline: Ingest_PL_Oracle * Updating linkedService: Ingest_LS_SQLServer_WinAuth * Updating linkedService: Ingest_LS_SQLServer_SQLAuth * Updating linkedService: Ingest_LS_SQLDB_SQLAuth * Update adf-cicd.yml adding additional parameters * Updating linkedService: Ingest_LS_Oracle_SIDAuth * Updating pipeline: Ingest_PL_Oracle * Updating linkedService: Ingest_LS_SQLServer_WinAuth * Updating linkedService: Ingest_LS_SQLServer_SQLAuth * Updating linkedService: Ingest_LS_SQLDB_SQLAuth * Updating pipeline: Ingest_PL_MSSQL * Update functions-cicd.yml * Adding integrationRuntime: SHIRcumulusDataSources02 * Updating linkedService: Ingest_LS_Oracle_SIDAuth * Updating linkedService: Ingest_LS_Oracle_SIDAuth
* Update SQLCMD variables * update keyvault secret name * Added demo connection variables * Update sql-server-common-cicd.yml * Update Control SQLCMDs and MetadataAsCode * updates to ingest cicd * remove unused notebook and update transform cicd * Creating Metadata.core and metadata.data projs * create yml files for core, data and master * Update sql-server-master-cicd.yml * Add Pipeline dependencies and move yml files to folder * Add transform BL Notebooks, metadata as code and integration execution in dev. * Add transform BL Notebooks, metadata as code and integration execution in dev * rename transform pipelines * Pipeline name tidy-up --------- Co-authored-by: MattPCollins <mc12338@my.bristol.ac.uk> Co-authored-by: MattPCollins <34279269+MattPCollins@users.noreply.github.com>
* Update notebook path * Update dim products notebook content.
* Update dimdate typing * Updating DAB config for historic files * Updating DAB config for historic files * Updating DAB config for historic files * Updating DAB config for historic files * Updating DAB config for historic files * Updating DAB config for historic files * Updating DAB config for historic files
* Update sql-server-master-cicd.yml * Update sql-server-core-cicd.yml * Update sql-server-core-cicd.yml * Update Script.PostDeployment.sql * Update Script.PostDeployment.sql * update stored procs to be permenent instead of temp * test original sql-server-core-cicd.yml with new stored procs * Update sql-server-core-cicd.yml
* Update README with Getting Started and Marketplace information.
* Add template for Deploy SQL PS scripts * SQL powershell deployment * Add entra admin + sql creation * Uncommenting * revert bicepparam values * Community Deploy scripts --------- Co-authored-by: CFToby-Andrew <toby.andrew@cloudformations.org>
* Adding linkedService: AmazonS31 * Updating linkedService: AmazonS31 * Adding Parquet: Ingest_DS_Amazon_S3_Parquet * Adding linkedService: Ingest_LS_AmazonS3_AKAuth * Updating Parquet: Ingest_DS_Amazon_S3_Parquet * Adding pipeline: Ingest_PL_AmazonS3 * Deleting linkedService: AmazonS31 * update dataset payload stored proc * update default connection types * Adding linkedService: Ingest_LS_DataLake_AKAuth * Updating linkedService: Ingest_LS_DataLake_AKAuth * Deleting linkedService: Ingest_LS_DataLake_AKAuth --------- Co-authored-by: MattPCollins <34279269+MattPCollins@users.noreply.github.com>
Adding pipeline: Wait 2
* updated stored proc * Updating pipeline: Transform_PL_Unmanaged --------- Co-authored-by: CFToby-Andrew <toby.andrew@cloudformations.org>
* Update GitIgnore and remove pyc files * Update Node Version for ADF GitHub Workflow * Deleting dataset: Ingest_DS_Oracle_SIDAuth * Updating pipeline: Ingest_PL_Oracle * Deleting linkedService: Ingest_LS_Oracle_SIDAuth
* Commit changes for new Data Visualisations via mermaid + CICD Deployment fixes for SQL * Adding extended mermaid pipeline for ADO targets * Update deployment setting + latest lineage sproc changes * Include lineage fine tune + deployment script tweaks * Amend validation of deployment * Updating Function App deployments * Enhance pipeline instance description
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.