forum New: opt-in AI analysis on your documents (and more editor improvements)
Started by @andrew health_and_safety flash_onAdmin
tune

people_alt 107 followers

@andrew health_and_safety flash_onAdmin

I'm so freaking excited about this update and I'm working on a full blog post explaining all the new features, but I wanted to give all the wonderful forumsfolk a heads up beforehand about what's new. I've been working on this feature for almost 6 months so it's super exciting to see it finally come to fruition.

Edit: Here's the full blog post! :)

Anyway, there's a ton of stuff packed in the update that went out last night. The two big AI features currently require a lot of processing power on the server (and one literally runs on a supercomputer at IBM) so they're currently only available to Premium users, but I'll be monitoring costs and figuring out ways to get them into more peoples' hands.

So here we go:

  • The document editor is now waaaaay prettier and runs snappier. The width of the document is now properly sized according to recommended online guidelines and the typography on headers, etc is now more reasonably styled. The editor should also work a bit better on phones and tablets, also.

  • You can now add your documents to a universe. You can also filter your document list by universe (like you can any other kind of page), and you'll find documents automatically hide themselves whenever you're using the "universe focus" feature at the top of the sidebar (that only shows pages from a singular universe). You can now also manage your document privacy settings per-universe (like you can other pages) – so if you add a document to a public universe, that document also automatically becomes public; marking that universe private will also mark all the pages within it as private, and so on.

  • You can now analyze your documents with AI to measure writing style and content. I'll post some screenshots in a second comment because I think this feature is too cool and want to show it off, but you can see a ton of useful information about your writing style, the characters in your story, their emotions, and how you stack up against some famous authors/pieces. Right now it's mostly a comparison to Hemingway and/or typical writing, but I'd love to use this analysis tech to analyze more authors so you can choose your favorite authors to compare your own writing to. :) This feature is entirely opt-in and no AI will run on your documents unless you request an analysis. This feature is only available to Premium users.

  • You can now add quick reference tabs to the document editor for any of your pages. These tabs appear on the right side of the screen and will open a sidebar containing all of that page's information whenever you click one. This lets you reference any answer to any field on any page while writing without having to leave the editor. Quick reference tabs are automatically created whenever you @mention a page, whenever the AI detects you're referring to one of your existing pages (during an analysis), or whenever you manually add a quick reference. This feature is only available to Premium users right now.

  • Adding a quick reference page to a document also creates a "Documents" tab on that content page. For example, if you add "Alice" to a couple documents she appears in, she'll also get a tab on her page that lists those documents.

Screenshots below, but – as always – if anyone has any feedback or suggestions on how to improve the document editor, the new document analysis, or the new quick reference feature, I'd love to hear them! :) This is a huge update with tons and tons of new code behind the scenes, so if anyone sees anything wonky anywhere, please do let me know!

Happy worldbuilding!

@Riorlyne pets

This looks amazing! I can’t wait to try it out. :D

How does all the analysis work with fantasy-style writing? Most writing gauges I’ve tried mark my writing as very difficult/university level, but I think it’s because all my fantasy names skew the common-to-uncommon words ratio. (The level drops waayyyy down when I change Tiere to Tina and Hals to Harry.) It’s hard to then get an accurate idea of the difficulty.

@andrew health_and_safety flash_onAdmin

@Riorlyne There's actually a bunch of different writing gauges out there that measure different things behind the scenes, and they're all good for measuring different things (and therefore not as good at measuring other things).

For example, the long names in fantasy novels might be disproportionately marked as harder to read by e.g. the Coleman-Liau index (which measures letters per word as a big metric) or the Flesch-Kincaid reading ease (which measures syllables-per-word). On the other hand, measurements like FORCAST only care about the number of one-syllable words compared to two-or-more-syllable words, so there's no difference between naming a character "Billy" and naming him "Biltaskothaster".

The analysis takes the five most popular of these readability metrics and combines them into a singular number. Since documents can belong to universes now and universes have a "genre" field, I should be able to improve things by adjusting weights for e.g. FORCAST to be weighted higher when the document is set in a fantasy or sci-fi world. Right now all the weights are (mostly) equal, but that's something that'll improve over time as more analyses happen.

A bit of a dream of mine is to also come up with my own readability formula (and be able to do things like excluding names altogether), but that's a bit far off. ;)

@Riorlyne pets

@andrew (Our Supreme Lord and Overseer) - Thanks for the detailed answer! I think the one where I kept getting marked disproportionately high was based on how many words weren’t in the X hundred that a 3rd grader should know (or something like that).

I’ve been trying it out, and I can’t click the explanation links (using Safari on iPad) - the ones that say “why is this important” and “how to use this chart”. Also, how are the emotions calculated? My character depressed over his mother’s imprisonment got marked as high in joy, and needless to say I’m really curious!

(If I’m asking questions that will already be answered in the long blog post, I’m happy to wait till that’s out!)

@andrew health_and_safety flash_onAdmin

Haha no, these are great questions. The blog post is up now (here!) but these specifics aren't in it. I'm going to try to add more of this kind of information to the analysis itself (for example, explaining what each reading metric measures and what is important to raise/lower if you want to affect that score).

The emotions are from a set of neural networks and classifiers run on IBM Watson. After running entity recognition on the writing (to pull out the character names and references), a large neural network looks at emotions and tone at a lexicon/word-level and relates them to the extracted entities with a lot of crazy statistics. Obviously, "emotion" is a super-broad field and hard to pick up even as a human, so the models Watson uses might not always be 100% accurate, but they are always improving little by little. I've had a few weird readings where a character's dominant emotion wasn't anything close to what I expected, but in one case it was interesting to read through with that context and see what positive words were used around that character, rather than by/toward that character itself. I changed the positive words to ones that matched the tone I intended for that character a bit better and I think it turned out better in the end. That might not always work out though. :)

I'll take a look at those explanation links on Safari/iPad; thanks for the heads up!

@ninja_violinist

This is so cool. I just let it analyse one of my documents and it was so much fun!! Thank you so much for setting this up!!
Where does it get the links to external references that it offers at the end? It told me to look up "blood stain pattern analysis" (fun but slightly concerning), so where would it search for stuff like that online and on what basis is it recommended?
Also - is there some way to communicate to the AI the pov character? I write in first person, so it only analysed the secondary character who was also in the scene (which was still really really cool).

@andrew health_and_safety flash_onAdmin

Ha, that's a complicated answer. But first let me preface that those reference links are probably the most experimental of all the analysis features.

In the most basic sense, it works primarily by looking at word choice and compares the words used to words used in labeled training data. For example, a training document about cars would probably talk about speed, color, doors, hoods, and so on. Obviously, those words could all apply to things other than cars on their own, so the neural network doing the classification looks at all the potential tags of all the words and makes its best guess by cross referencing all the topics that every known word could relate to and measuring how likely that topic is based on how many words (potentially) reference it. I have a confidence threshold that I discard guesses under (so if the AI isn't at least e.g. 70% certain a topic applies, I ignore it). Finding the proper confidence level that minimizes incorrect predictions without excluding correct ones is what I can (easily) adjust here, so I'm watching for feedback on these features to see if I need to adjust that confidence threshold up (if guesses are wrong) or down (if no guesses or not enough guesses).

I'm not really familiar with blood stain analysis myself (ha), but I'd guess something in the word choices in your story is similar enough to the word choices used in blood stain pattern analysis. Neural networks are great for classification problems like this, but unfortunately they're not good at all at describing why they classified something. It's our job as humans to make the call on whether those classifications are right or not; sometimes they are and sometimes they get just a little confused. :)

@andrew health_and_safety flash_onAdmin

@ninja_violinist There isn't a way to analyze a first person narrator yet. The most exciting part of this tech (in my opinion) is that it's literally bleeding edge work in a new frontier. That's a double edge sword though: it usually means no one has done X or Y before, which means I get to do it myself – which also means it's incredibly hard and will probably take a significant amount of time. I appreciate the patience and shared excitement though. :D

@Riorlyne pets

@andrew (Our Supreme Lord and Overseer) - What about analysing different nicknames as the same person? I tested it on a longish piece where the same character was called by their full name, a nickname, and their title (e.g., Timothy, Tim, the blacksmith), and the analysis gave me a rundown for three separate characters. This character wasn’t in my character pages, but if I had one for Timothy, with “Tim” and “the blacksmith” listed under nicknames, would that make it more likely that multiple nicknames be recognised as the same person?

@andrew health_and_safety flash_onAdmin

I'm researching the best way to analyze and merge recognized entities now. What you can do if you do have a Tim page is to click "Link page" on each of the different nicknames and select your Tim page. That should link them all together and have them all treated as a singular person – Tim.

Deleted user

I knooooooooooooooooooooooow! I've already used the new features. It's really cool to see the breakdown. Although, I have now become aware that many of my works have a 'melancholy' feel to them. So word choice is super important for me now. (awkward laugh)