What is ideal for parsing Obsidian markdown?

[Feel-ix-343]

@dawsers

The hardest part of an LSP server is probably parsing. Regular expressions can become very long and complicated. There are very subtle bugs that can creep every now and then, making the code very hard to maintain. I had a look at the code, and it seems that is the place where most of the future hard work will be: capturing code blocks, complicated links, etc. I see sQVe is helping you debug those.

The reason why I am making this comment is I had an idea I wanted to share here. you could use tree sitter for parsing. There is a quite mature implementation of the markdown parser, and there are Rust bindings for tree sitter. It this was a C++ project I would definitely help you, but I am quite illiterate when it comes to Rust. I could help you with queries for grouping, semantic tokens etc, though. I have my own version of those for markdown. The advantage of this is you could decouple some of the nastiest bugs from your project, and if you found bugs, help them fix them if you have the time, but without the same level of commitment. This adds a dependency on others, but at the same time you get a lot of work already done "for free", and if any of their decisions don't suit you, you fork from an advanced state.

It is just an idea.

@Feel-ix-343

Great comment!

So originally I started by using the markdown treesitter parser, but the issue was that with my vault of notes (4K files), treesitter was extremely slow. Parsing synchronously took around 2.3 seconds and while parsing async (on a 20 thread CPU) took 657ms. Now on my vault regex parsing is near instant (less than 5ms I believe). The other thing is that the parser is not obsidian specific; it will get the general markdown stuff (links ...), but not the specifics like the obsidian query language, dataview, and the obsidian yaml -- which I want to eventually support (I'm very excited about giving query/dataview completions and calculating/rendering queries for find_references and hover!).

However, I do definitely agree that regex is not ideal. There are several glaring issues:

As the rust regex package does not support recursive parsing, it is impossible (to my knowledge?) to parse links that have links (that have links...) in them. For example, there are times when we use links in heading names # heading [[link]]; a link to this -- if in file.md -- would be [[file#heading [[link]]]]; Given that there could be infinite levels of nesting, I cannot guarantee that all links in links will be parsed; right now, the regex will not even parse [[file#heading [[link]]]]. For now, I will probably work around this by handling one or two levels of links in links.
The regex to parse links (probably not ideal; but kinda intuitive) is this: \[\[(?<filepath>[^\[\]\|\.\#]+)(\#(?<infileref>[^\[\]\.\|]+))?(\|(?<display>[^\[\]\.\|]+))?\]\] (which is for the [rust regex](https://docs.rs/regex/latest/regex/) syntax). This is unbearably ugly and offputting!

Anyway, right now it works and it is fast (very important 💯 ), but real parsing is certainly on the future path of this project. The options I am considering are

Using treesitter in combination with regex to get feature-rich obsidian support (thank you @dawsers for mentioning this)
Writing a recursive parser using some rust parsing library ([chumsky](https://github.com/zesterer/chumsky) and [nom](https://github.com/rust-bakery/nom) seem appealing). The perfect library would allow me to reuse some other person's markdown parsing code, extending it with obsidian features.
Making the regex better and/or using a recursive regex parser
... [any suggestions?]

Again, thank you for the comment; I am am quite excited that this project is gaining some momentum -- my confidence is boosted a bit from this. I say: Lets make the best markdown LS for PKM in existence!

@dawsers

Fair enough. I didn't know the history of your project.

However, the current parser isn't as full featured yet. The markdown tree sitter one has to do two passes to support code blocks etc., and markdown-oxide doesn't support those yet. I am sure you will be able to write something very fast, but it won't be as fast as it is now, in part because of the issues you mention, and in part because you will still need to add more passes to support all the features. On the other hand, 4000 notes in less than one second...where do I sign?! :-) That is just startup cost to build the hash structures for the vault, references afterwards update instantly, and added/modified files only need to update partially, I guess. For most people that is not an issue, any LSP server for programming languages takes a lot longer to index a project and provide semantic highlighting.

Thank you again for your dedication. I was just trying to think about how we could make your life easier, of course.

[Feel-ix-343]

The current options:

1. Regex: Easy(ish) and fast
2. Treesitter: Easy(possibly) but slower initially but fast subsequently
3. Other recursive parser: Hard and undetermined speed

Which is best?

[EphemeralDev]

Have you looked at pomsky before? I don't know if it will help with speed at all but seems like it should help with the ugliness/maintainability aspect of the regex. Bonus points is that it's also rust, so hopefully it wouldn't complicate things too much.
Website and the github repo.

[Feel-ix-343]

Great I will consider it. Seeing that it will compile to regex, however, I don't think it will fulfill the project's needs for recursive parsing.

However, if creating our own parser or using treesitter is far to slow or difficult, this may be a good option.

[Feel-ix-343]

Seems that I am going to use tree-sitter for version 1.