[API Change] Support bytestring custom writers.

New-style custom Lua writers can now define an alternative entry function
`BinaryWriter`. If a function with that name is defined, then pandoc
will treat the returned string as binary output. This allows to generate
formats like docx and odt with custom writers.

2 files changed, 36 insertions, 14 deletions

diff --git a/doc/custom-writers.md b/doc/custom-writers.md
index a6e657804..0b1458857 100644
--- a/doc/custom-writers.md
+++ b/doc/custom-writers.md
@@ -16,10 +16,12 @@ install any additional software to do this.
 [Lua]: https://www.lua.org
 
 A custom writer is a Lua file that defines how to render the
-document. Writers must define just a single function named
-`Writer`, which gets passed the document and writer options, and
-then handles the conversion of the document, rendering it into a
-string. This interface was introduced in pandoc 2.17.2.
+document. Writers must define just a single function, named either
+`Writer` or `ByteStringWriter`, which gets passed the document and
+writer options, and then handles the conversion of the document,
+rendering it into a string. This interface was introduced in
+pandoc 2.17.2, with ByteString writers becoming available in
+pandoc 3.0.
 
 Pandoc also supports "classic" custom writers, where a Lua
 function must be defined for each AST element type. Classic style
@@ -29,9 +31,9 @@ writers if possible.
 # Writers
 
 Custom writers using the new style must contain a global function
-named `Writer`. Pandoc calls this function with the document and
-writer options as arguments, and expects the function to return a
-UTF-8 encoded string.
+named `Writer` or `ByteStringWriter`. Pandoc calls this function
+with the document and writer options as arguments, and expects the
+function to return a UTF-8 encoded string.
 
 ``` lua
 function Writer (doc, opts)
@@ -39,6 +41,14 @@ function Writer (doc, opts)
 end
 ```
 
+Writers that do not return text but binary data should define a
+function with name `ByteStringWriter` instead. The function must
+still return a string, but it does not have to be UTF-8 encoded
+and can contain arbitrary binary data.
+
+If both `Writer` and `ByteStringWriter` functions are defined,
+then only the `Writer` function will be used.
+
 ## Example: modified Markdown writer
 
 Writers have access to all modules described in the [Lua filters
diff --git a/pandoc-lua-engine/src/Text/Pandoc/Lua/Writer.hs b/pandoc-lua-engine/src/Text/Pandoc/Lua/Writer.hs
index 8dbae9eae..f2d75b905 100644
--- a/pandoc-lua-engine/src/Text/Pandoc/Lua/Writer.hs
+++ b/pandoc-lua-engine/src/Text/Pandoc/Lua/Writer.hs
@@ -56,13 +56,25 @@ writeCustom luaFile = do
     let writerField = "PANDOC Writer function"
 
     rawgetglobal "Writer" >>= \case
-      TypeNil -> do
-        -- Neither `Writer` nor `BinaryWriter` are defined. Try to
-        -- use the file as a classic writer.
-        pop 1  -- remove nil
-        return . TextWriter $ \opts doc ->
-          liftIO $ withGCManagedState luaState $ do
-            Classic.runCustom @PandocError opts doc
+      TypeNil -> rawgetglobal "ByteStringWriter" >>= \case
+        TypeNil -> do
+          -- Neither `Writer` nor `BinaryWriter` are defined. Try to
+          -- use the file as a classic writer.
+          pop 1  -- remove nil
+          return . TextWriter $ \opts doc ->
+            liftIO $ withGCManagedState luaState $ do
+              Classic.runCustom @PandocError opts doc
+        _ -> do
+          -- Binary writer. Writer function is on top of the stack.
+          setfield registryindex writerField
+          return . ByteStringWriter $ \opts doc ->
+            -- Call writer with document and writer options as arguments.
+            liftIO $ withGCManagedState luaState $ do
+              getfield registryindex writerField
+              push doc
+              push opts
+              callTrace 2 1
+              forcePeek @PandocError $ peekLazyByteString top
       _ -> do
         -- New-type text writer. Writer function is on top of the stack.
         setfield registryindex writerField