Style guides vary, but I recommend "dialogue" for conversation and "dialog" for a digital "dialog box."
Last week, when I was writing about whether you should use italics or quotation marks for internal dialogue, I was reminded that people sometimes aren’t sure how to spell the word. Is it D-I-A-L-O-G-U-E or simply D-I-A-L-O-G? I see both, and I even had a writer alternate between the two in a recent draft I was editing. I had to fix that!
‘Dialog’ and ‘Dialogue’ in British English
‘Dialog’ and ‘Dialogue’ in American English
The shorter spelling started spiking in the early 1980s, especially in American English, and in many cases it’s used when people are writing about computers—for example, when people are writing about a dialog box where you input text. That’s often spelled D-I-A-L-O-G.
It seems that the computing use—the shorter spelling—again, especially in American English, has started drifting over into the older “those two people are talking” meaning.
Nevertheless, the AP Stylebook recommends the longer “dialogue” spelling even when you are writing about a dialog box.
The Chicago Manual of Style doesn’t specifically recommend one spelling over the other, and it uses the longer spelling throughout its own book, but in an FAQ entry says that the shorter spelling is fine when you’re writing about a dialog box.
Use ‘dialogue’ for a conversation and ‘dialog’ for a digital text-entry box.
I favor Chicago’s approach in this case, so my recommendation is to use the longer spelling when you are writing about conversations in a novel or play and use the shorter spelling in the phrase “dialog box” since that is what seems to be common in the computer industry.