Enums/Flags

[Marlamin]

We've been talking about enums/flags support in WoWDBDefs for years now and haven't actually gone forward with it and it's probably time to change that.

Shipping it as part of a 'V2' of the DBD format might be a bit much to take on or wait for, so a different strategy was discussed in the datamining Discord that I want to bring over here because having important discussions like that on Discord sucks.

Stuff that was previously discussed in the V2 thread on here as well as more recently on Discord:

File structure

• Using separate files to describe flags/enums as well as link them to fields as to not require any changes in the DBD files themselves.
• These files could live in an extra folder.
• A mapping file could be used to 'map' columns inside of DBD files to their respective enums/flag definitions.
• Files named after their flag/enum would live next to the mapping file.

Format

• Similar to DBD? Different? Formats were previously suggested in the v2 thread and in schlumpf's demo definition format.
• Having flags/enums described in hex was preferable, is that still the case?
• Do we want to include longer descriptions of enums/flags? Would they be comments or part of the actual format? I'd love to have this for user-facing tool/diff reasons.

Versioning

• How do we tackle this?
• Do we need to? Are there tangible examples of Blizzard changing flags/enums over time to the point where one would need to redefine them similar to how we redefine version definitions in DBDs?

Would love to get some discussion going and maybe even work towards an actual implementation. As we're doing it separately from the fabled 'V2' format this can be done without disturbing the current definitions.

[QartemisT]

My main comment on this would be being able to provide a version range on some of them, as they change over time. :3

[Marlamin]

Do we only need versioning for the enums/flag files themselves and not the mapping file? Have enums/flags every switched meanings for certain fields entirely?

[bloerwald]

I sure agree on "no complete copies of the list on every minor addition". It might make sense to allow the entire things to change so much to warrant an entirely new list though.

A few independent thoughts:

[C] As a person adding information, I don't want to be forced to define a version, especially not if it has to be there and has to be exact. I don't have the energy to bisect client decompilation to figure out when code was added to handle it.

[A] What about a hybrid format where sub ranges can be version-specific, and overwritten?

enum "M2 Attachement" {
  0 Shield
  0 MountMain
  0 ItemVisual0             // this is a side-comment: allow multiple names for same value, please
  1 HandRight
  1 ItemVisual1
  …
  38 SpellHandDirected
  
  version ≥ wotlk {
    39 VehicleSeat1
    40 VehicleSeat2
  }

  version ≥ Mists {
    55 HeadTop
  }

  version ≥ TwilightFalls {
    38 SpellFeetPorn
  }
}

[B] If stuff changed that much, is it really still the same field/type/enum? We already had this pain with the SoundEntries → SoundKit change everywhere, and I think we just plain ignored it, keeping field name, and changing referenced table to the new one, breaking all old definitions. The cleaner solution would have been adding a new field with separate name and fk definition and using that instead. A more painful solution, but a cleaner one. This essentially is an argument for "decouple enum definition and mapping, and allow versioning it". It even expands to "fk mapping should be outside as well".

[Marlamin]

C: Agreed.
A: Inline versioning, while far more readable for humans, worries me in terms of format parsing complexity. I don't have a better solution, but I know this has been a pain point (for the in my opinion rather simple DBD format) in the past, so something to take note of at least.
B: There's probably cases where it could be either the same field/type/enum but changed significantly or a different one entirely. And yes, SoundEntries -> SoundKit still pains me to this day. Adding FK mappings to the mapping/extra file is perfectly fine by me as a term of "overriding" FKs in DBDs. It would get rid of one the things we'd need to "fix" for V2 as well, so that's nice.

[bloerwald]

A: if there is a agreed-upon need to avoid a custom format and only have a custom schema, I’m fine with transitioning to YAML. I had expected the custom format to be easier, but I‘m not a user.

[Marlamin]

Its more nuanced than just "no custom format", just not too complex of one. Again, I don't have a solution, an easily already parseable format is one, an easier custom one without too many shenanigans is another.

[Dorovon]

A few special cases that I think we should try to support in the format include

• Conditional Enums/Flags/FKs: One option here would be to have a section of the mapping file specifically for these fields where instead of listing a field and its enum name, it instead lists the field, the conditional field, and the required value of that field before the enum name.
• Empty Flags: This would be for fields that we know are bitfields, but where we don't know what any of the flags are so that tools can still display them as flags. These could be defined alongside other fields, but simply without a reference to a flags file.
• FK Flags: For example, SkillLineAbility::RaceMask appears to use ChrRaces::PlayableRaceBit as flags.

Long Descriptions

• Do we want to include longer descriptions of enums/flags? Would they be comments or part of the actual format? I'd love to have this for user-facing tool/diff reasons.

I think this is a good idea, but we'll have to decide how we want to format the short and long strings together. One of the big use cases for having these files is for tools like DB2 viewers where being able to read more about what an enum value does would be very helpful. There is also the question of just how detailed/long we want to let these be. As an example for spell effect types, we could have a brief descriptions to give a simple idea of what something does or long descriptions that fully explain what the effect does and how the spell data is used.

98 "Knockback" "Knocks the target(s) in a direction."

98 "Knockback" "This overrides the current velocity of the target(s). Vertical velocity is set to EffectBasePointsF/10 yards/sec up and horizontal velocity is set to EffectMiscValue[0]/10 yards/sec away from the origin of the Knockback."

The former is short enough to show in most text diffs but isn't very useful in this case, while the latter is long enough that it wouldn't display very well in some cases but provides most of the useful details someone reading about a spell may want to know.

Types?

While we're at it, I think it could be helpful to also use the mapping file to map fields to generic types that show up a lot where we would want our tools to handle all of them in the same way. The biggest example that comes to mind here is the integer fields that represent colors used in many DB2s, but we could also use this for Unix timestamps and anything else sensible that shows up.

[Marlamin]

Yeah, for the mapping we could do like...

FLAGS Achievement::Flags AchievementFlags
FLAGS SpellMisc::Flags[0] SpellAttributes0
FLAGS SpellMisc::Flags[15]
ENUM Map::ExpansionID ExpansionLevels
COLOR LightData::SkySmogColor

Where SpellMisc::Flags[15] is marked as a flags field but doesn't have an accompanying file yet.

Conditionals have a third argument to specify their condition, can also make them CENUM/CFLAGS or CONDENUM/CONDFLAGS for additional clarity.

ENUM ModifierTree::Asset ItemQuality ModifierTree::Type=14 
FLAGS ChrCustomizationReq::ReqValue ClassMask ChrCustomizationReq::ReqType=1

For descriptions for enums themselves we could have the really long ones/explanations be a comment which is still accessible to parsers.

98 KnockBack "Knocks the target(s) into a direction" // This overrides the current velocity of the target(s). Vertical velocity is set to EffectBasePointsF/10 yards/sec up and horizontal velocity is set to EffectMiscValue[0]/10 yards/sec away from the origin of the Knockback.

FK flags... hmmm... maybe just another :: field instead of a flag name? Unsure.

[Dorovon]

Thinking about it more, maybe FK Flags aren't such a huge deal unless I'm forgetting other places where they get used. For cases like RaceMasks most tools would probably just want to show the name anyway, so we could just do those manually.

[bloerwald]

I absolutely loved the wow.tools content dependent fkeys etc. I do support this. It is another reason for me to pivot to more complex FK definitions and making those conditional and versioned.

Empty Flags

Fine with me. Just as fine as adding placeholder flag definitions though. Both with "seen values" added (I had done that in comments in the past, even without knowing semantics) and with a completely empty list.

FK as Flags

Not a huge deal, but why not. It is a trivial annotation on FKeys. int<ChrRaces::PlayableRaceBit$Mask>?

Long Descriptions

I think that's why I added COMMENT everywhere back then. There is a need for developer-readable text which isn't displayed on first sight in UIs.

[Ovahlord]

Joining in to give some further examples which underline the usefulness of adding version specific flags as I was asked to name some clear examples for it here:

SpellEffects.dbc/db2 is the most apparent candidate which would benefit from version specific enums as the earlier expansions (WoD and onwards) started to deprecate old effects and re-use them for new effects.

Classic was confirming this further by re-introducing some of the deprecated spell effects under new enum values which of course do not line up with older versions.

One example would be SPELL_EFFECT_ACTIVATE_RUNE. That effect was introduced with original Cataclysm (4.0.1) and later deprecated and replaced for SPELL_EFFECT_RESTORE_GARRISON_TROOP_VITALITY (enum Id 146)
However, Cataclysm Classic re-introduced the effect under enum Id 330 which implies that the enums do not align anymore and need version specific definitions. This applies to quite a few effects

So the question:

Do we need to? Are there tangible examples of Blizzard changing flags/enums over time to the point where one would need to redefine them similar to how we redefine version definitions in DBDs?

Is a very yes from my side. There are enums which have changed. The versioning is a tough question though because no matter how you twist it, it's a balance between either doing entire version-specific files (which is verbose as hell) or doing unreadable inline modifications.

[Marlamin]

I think that settles the need for the version-specific definitions then. Just a matter of figuring out the best format for them.

[Marlamin]

Given there's been no movement on this I'm going to bite the bullet and get it going. Still not a clue how to handle version-specific definitions in a way that isn't ugly or verbose, but whatever.

[Dorovon]

For versioning, I wouldn't mind something similar to DBDs where we just duplicate the whole list, although I think we'd also need an option to more loosely match versions since unlike DBDs I imagine these enum definitions will not be automatically updated and some enums may go years where nobody touches them. In most cases, I think all we're really going to need is a mainline and a classic era list, although the classic era list would probably need to have some way to define a cutoff since I think over time Blizzard will be merging a lot of them to be the same as mainline to make their lives easier. They may have already done this in some places; for example: mainline has no spells with SpellEffect::EffectAura values in the [546, 633] range.

I suppose a simple way to loosely match versions without requiring updates ever would be having some magic keywords somewhere like "MAINLINE" and "CLASSIC_ERA" that can be used to pick a list when there is no good match for a build.

[Marlamin]

For versioning, even though I'm not happy with this at all, went with this for now to keep it similar to DBDs (partial SpellEffect::Effect enum):

145 PULL_TOWARDS_DEST
(BUILD 3.0.1.8303-3.3.5.12340, 4.0.0.11792-4.3.4.15595, 5.0.1.15464-5.4.8.18414) 146 ACTIVATE_RUNE
146 RESTORE_GARRISON_TROOP_VITALITY
147 QUEST_FAIL

We may also want to adopt EXPANSION (or even BRANCH as Dorovon mentioned) filters to make the lines less long. These would be internally set and replaced with an applicable build range/set of build ranges, which would then work the same as existing build ranges, e.g.

(EXPANSION WotLK, Cata, MoP) would be equivalent to (BUILD 3.0.1.8303-3.3.5.12340, 4.0.0.11792-4.3.4.15595, 5.0.1.15464-5.4.8.18414).

This should only be used to note exceptions. The default mapping for the value, which should be the most recent one, as such has no build notation or anything like that. You'll note that e.g. (Non-Classic) Vanilla/TBC aren't noted there because it didn't have runes, so it wouldn't have this enum value regardless.

Implementation wise this means a preferred build or "build hint" has to be given to the parser for DBDE/DBDF files. If no build is given, it will ignore any build-specific values, if a build is specified it will prefer lines with build-specific values that include this build, but uses the default value otherwise.

I didn't want to do full duplicate lists like DBD does as that would be too annoying to maintain I think, but more feedback on this is still welcome and we can still change it while adoption is non-existent/very small.

[Marlamin]

The initial pass at converting ~500 enums and flags to the new format is pretty much done, largely sourced from WTL and wago that have in turn sourced them largely from TC/the wiki/enumeratedstring.db2. I'm sure many things still need updating, so we're probably ready to start accepting any PRs and issues around the enum/flag contents now.

Implementation wise, DBDefsLib integration is done outside of more tweaks/testing that need doing and DBCD support still needs some work (see the open PR). The upcoming version of wow.tools.local will be using these (but the enum-conversion bits of DBCD may remain disabled for WTL).

The floor is still open to suggestions for tweaks to the format and such, but I'd like to have this first stab at the format and implementations finalized in the next week. Format and implementation changes will still be possible afterwards as going off the amount of activity here, the amount of affected users will be pretty small.

Still not the happiest with the format, but happy to finally have these in the repo. Thanks for everyone's contributions so far.