Really interested in this mod but I just wanted to know before I buy, how well does this mod work with custom bbcodes?

Well it actually depends on what do you mean by custom bbcodes, there is a guide (available to customers) to integrate a new button in less than 2 minutes; however that only adds the button to the editor, if you actually want to add a WYSIWYG (for example, to actually view the video on the editor if you're talking about a YouTube BBC) view of the bbcode then it demands much more code (but it's still possible).

I'm interested in both since a lot of my users want something they can see as they edit. How difficult is it to do the latter and is there a guide for me to do it?

There is no guide and it's just a little bit easier than doing the same at the default SMF editor. This mod is indeed much more user friendly than the default editor, but in terms of programming your own WYSIWYG views it still requires some knowledge and there is no guide at the moment to achieve that.

Thanks! Good to know.