Extract integration testing utils and runner

[Ethan-Arrowood]

• extract integration test utils to @harperfast/integration-testing-framework
• fix: use file: reference for integration-testing-framework devDependency
• actually get it to work
• migration to published @harperfast/integration-testing package

[socket-security]

Caution

Review the following alerts detected in dependencies.

According to your organization's Security Policy, you must resolve all "Block" alerts before proceeding. It is recommended to resolve "Warn" alerts too. Learn more about Socket for GitHub.

Action	Severity	Alert  (click "▶" to expand/collapse)
Block		HTTP dependency: npm hdd-space depends on https://github.com/int0h/parse-columns.git Dependency: parse-columns@https://github.com/int0h/parse-columns.git Location: Package overview From: package-lock.json → npm/@harperfast/integration-testing@0.2.0 → npm/hdd-space@1.2.0 ℹ Read more on: This package | This alert | What are http dependencies? Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev. Suggestion: Publish the HTTP URL dependency to a public or private package repository and consume it from there. Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore npm/hdd-space@1.2.0. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.
Warn		Low adoption: npm harper Location: Package overview From: package-lock.json → npm/@harperfast/integration-testing@0.2.0 → npm/harper@5.0.0 ℹ Read more on: This package | This alert | What are unpopular packages? Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev. Suggestion: Unpopular packages may have less maintenance and contain other problems. Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore npm/harper@5.0.0. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.
Warn		Low adoption: npm parse-columns Location: Package overview From: package-lock.json → npm/@harperfast/integration-testing@0.2.0 → npm/parse-columns@1.3.0 ℹ Read more on: This package | This alert | What are unpopular packages? Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev. Suggestion: Unpopular packages may have less maintenance and contain other problems. Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore npm/parse-columns@1.3.0. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

View full report

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	@​harperfast/​integration-testing@​0.2.0

View full report

[kriszyp]

I'm curious why we don't just export these modules (add them to the exports of the package.json)? If you already have harper installed, why not access them that way instead of installing a separate package? It seems like the existing mechanism has a more reliable way of finding the correct/current harper starting module; is https://github.com/HarperFast/integration-testing/blob/main/src/harperLifecycle.ts#L146 going to reliably resolve to the correct module?
If we do it this, presumably we would need to update harper-pro as well, as it uses these integration test modules extensively (although I guess, I'm concerned that it would actually resolve correctly).

[Ethan-Arrowood]

The main reason I went this way was because if this was part of harper then it'd be subject to the semver of it. I know that test modules (Jest, React Testing Library, Vitest) tend to have more breaking changes than other things and I wouldn't want to hold us back by keeping it in Harper

[Ethan-Arrowood]

Plus, this way, users can really bring their own Harper installation. It can be built from source, or npm download, or something else! Embedding it in harper limits the usability in that way.

For @harperfast/nextjs, it doesn't actually depend on Harper (like harper is only a devDependency of the module). So in CI, once the plugin is built it, I can have a reasonably simple workflow matrix configuration to test the plugin across different Harper versions (harper, harper-pro, eventual future majors, maybe even figuring out backwards compat for v4??). This might make more sense once I actually finish that work and show (not tell)

[kriszyp]

Plus, this way, users can really bring their own Harper installation. It can be built from source, or npm download, or something else!

Isn't that exactly what adding the test modules to the exports would provide (although hard to evaluate "something else")?

My main concern is that we have still been actively making a lot of changes, fixes, and modifications to the integration testing modules in coordination with integration tests themselves. And this has already proven onerous to coordinate across repos, as we have experienced broken integration tests in harper-pro due to changes in the base modules (that weren't noticed as breaking due to the integration tests failing in a different repo). Creating another separate repo/package seems like it exacerbates this complexity and coordination challenge.

Anyway, I think this change might have been a bit premature and could have been planned further out. But I'm sure we can manage and make this work.

[kriszyp]

Also before merging, I believe we need to ensure there is a corresponding PR for harper-pro to update its integration test imports as well.

[DavidCockerill]

Interesting change, feels like it might add another level of complexity to test-based dev in Harper. I quite like the “everything is in one workspace” convenience. Makes debugging Harper really easy when running the tests. I’m not opposed to the direction, just curious how this will impact the current workflow.

[Ethan-Arrowood]

Isn't that exactly what adding the test modules to the exports would provide (although hard to evaluate "something else")?

My understanding is that if we had this included with the Harper module, then users will be stuck with the integration testing utils as defined at the specific Harper version.

By splitting it out, now we can freely iterate on either thing as much as we want, and users can get those changes without having to deal with updating the other one.

I don't really see how this adds another level of complexity; its the exact same code as before. npm link works great for local iteration.

We don't have to go this direction though if everyone feels strongly. I really only pulled it out because it did make my development experience better. If I'm on my own in that then so be it! I'm happy to continue with it in core.