AnnoGram

IEEE VIS 2025

AnnoGram An Annotative Grammar of Graphics Extension

Md Dilshadur Rahman*, Md Rahat-uz-Zaman*, Andrew McNutt, and Paul Rosen equal contribution

Paper Playground Cite

Make annotation part of the grammar

Popular charting libraries share one blueprint, the Grammar of Graphics. Annotations were never part of it, so they are drawn on top of the chart and break when it changes. AnnoGram adds them to the blueprint itself.

The blueprint

Grammar of Graphics

  • Data
  • Scales
  • Geometries
  • Coordinates
  • Guides
  • Annotations
  • ggplot2Rshares the blueprint
  • plotninePythonshares the blueprint
  • Vega-LiteJSONAnnoGram prototype
  • AltairPythonshares the blueprint
  • G2JavaScriptshares the blueprint

TodayDrawn on top, in pixels

AnnoGramDeclared in the grammar

Both notes start on the hybrid. When the chart changes, the note drawn on top stays where it was put; the note in the grammar finds the hybrid again.

ggplot2, plotnine, Vega-Lite, Altair, and G2 each trace their design to the Grammar of Graphics (Altair through Vega-Lite). The AnnoGram prototype extends Vega-Lite. Charts on this page use made-up values.

Every annotation answers three questions

What it points at, what it draws, and where it goes. Like a CSS rule, each annotation pairs a target with its effects. These three colors follow you through the rest of the page.

  1. 1What is being annotated?

    Target

    The specific visualization element an annotation applies to. It can be part of the chart, a rule over the data, a fixed position, or nothing at all.

    • Data points: individual points, or subsets chosen by direct indexing or by an expression, such as every value above a temperature threshold, updating as the data changes.
    • Axes: the x- or y-axis, to mark significant values, ranges, or tick marks, gridlines, and labels.
    • Chart parts: titles, legends, and axis labels, for meta-level explanations such as what a legend's colors mean.
    • None: a standalone note, placed independently to offer supplementary context.
  2. 2How should it be annotated?

    Annotation type

    The grammar accommodates all seven types of Rahman et al.'s taxonomy. These four are the ones the proof-of-concept implements.

    • Text

      Explanatory context for data points, regions, or encodings.

    • Enclosure

      Rectangles, ellipses, brackets, or braces around regions of interest.

    • Connector

      Paths or lines linking elements, such as a note and its point.

    • Indicator

      Lines, arrows, or highlights marking thresholds, trends, or events.

  3. 3Where does it go?

    Placement

    Annotations receive meaningful placement automatically from their target and the layout. Authors can override the defaults, the way inline CSS takes precedence over a stylesheet.

    • Fixed positioningIn data space, anchored to the encoding scales so it carries across chart types; or in pixel space, for absolute control. Fine adjustments use dx and dy offsets.
    • Relative positioning and anchoringAligns an annotation with its target as the chart changes, in one or both dimensions, snapping to the target's edges or center.

Annotations can work together

  • Reference

    One annotation explicitly links to another through its id. A marker on an anomaly, for instance, can refer to a separate annotation that explains its cause.

  • Composite

    Several annotations grouped together to describe a repeated structure in the data, such as seasonal trends across several time intervals, drawn with coordinated shapes, markers, and labels.

Both reflect annotation ensembles: annotations combined on purpose to convey meaning that a single annotation cannot express on its own.

On a real chart

A Washington Post chart of COVID-19 cases, recreated with the Vega-Lite Annotation prototype. Point at either annotation in the specification to find it on the chart.

"annotations": [ … ] Excerpt
Recreated with the prototype: vl-annotation.netlify.app/teaser
  1. An enclosure with a label. The gray band is an enclosure on a data-space target at 2021-01-11, and its text, “Jan. 2021 peak”, sits at upperMiddle.
  2. A text with a connector. The note targets the datum dated 2021-01-11, chosen by an expression, and a connector with a natural curve ties the two together.

The whole grammar

Every rule on one page, colored by family. Hover a name to trace where it is defined and used, or select it to read what it does.

keyword primitive
  • := ( | | | | | )[]
  • text :=opt ⟨⟩
  • enclosure :=opt ⟨⟩
  • connector :=opt ⟨⟩
  • indicator :=opt ⟨⟩

Annotation Targets

  • := Expression | index[]
  • := title | legend | subtitle | caption | ···
  • := axis := x | y, parts := label | tick | grid | tick-label range :=opt [] | Expression

Annotation Types

  • ⟨X⟩ := (id, , X),  = ···
  • := (string, | None)
  • := (Rect | Ellipse | SVGShapePath | ···, | None)
  • := (Markers | None, SVGShapePath, linear | catmull-rom | ···)
  • := (line | area | arrow | ···, Expression)

Placement

  • := | |
  • := auto | start | mid | end
  • := auto | upLeft | midRight | ···
  • := {type := data | pixel, x := , y := }
  • := string | number

Hover a name to trace where it is defined and used; select it to read more. Esc clears.

Try it: change the chart, keep the annotation

Switch the chart, update the data, or narrow the layout. The annotation is written once and found again every time; nothing in it has to change.

Target
Resolved to
Placement
Effects
Position
Specification chartannotation 

            

            

          

        

        
Made-up values, drawn by this page with a simplified specification. The real
          prototype runs in the playground.


      

    

    
    

      

        

          
How it works

          
The prototype, Vega-Lite Annotation, adds five steps to Vega-Lite's compiler
            instead of replacing it, so every chart still renders through Vega as usual.

        

        
    
          
  1. 
            1
            
    Parse
    
            
    Read the specification and pull out the annotations, at the top level or inside layers.
    
          
    2. 
          
  2. 
            2
            
    Resolve positions
    
            
    Find each target in the drawn chart and look for free space near it, unless the author fixed a
              position.
    
          
    3. 
          
  3. 
            3
            
    Assemble
    
            
    Link related pieces, such as a connector from a label to its enclosure, and resolve references
              between annotations.
    
          
    4. 
          
  4. 
            4
            
    Transpile
    
            
    Turn each annotation into ordinary Vega data and marks, beside the chart's own.
    
          
    5. 
          
  5. 
            5
            
    Post-add
    
            
    Add the few shapes that need special rendering, such as ellipses, straight into the drawn scene.
    
          
    6. 
        

        
Steps 2 and 3 repeat, backtracking to reduce overlap between annotations and marks.

      

    


    
    

      

        

          
How it compares

          
The same annotated bar chart, line chart, and scatterplot, built with the
            prototype and with eight other tools. The examples are on
            GitHub.

        


        

          

            

              

                
Lines of code for just the annotation

                
Annotated bar chart, from Table 1

              

              

                
                
                
              

            

            
    
            
    
            
    Fewer is better. Visual editors such as PowerPoint and Figma have no code, so they
              are not shown.
    
          
    

          
    
            
    Four qualitative dimensions
    
            
      
              
    • 
                
      1st-class annotation support
      
                
      Whether annotations are structured components within the visualization model rather than
                  external overlays.
      
              
      • 
              
    • 
                
      Intuitiveness
      
                
      How naturally the annotation syntax reflects the data and chart structure, and how easily common
                  annotation tasks can be expressed.
      
              
      • 
              
    • 
                
      Error proneness
      
                
      The likelihood of mistakes while authoring or editing annotations, such as misalignment or broken
                  references.
      
              
      • 
              
    • 
                
      Portability
      
                
      How well annotations adapt to different datasets or chart types without manual rework.
      
              
      • 
            
    
          
    
        
    

        
      
          
    • 
            
      Without annotation constructs
      
            
      D3, Vega, and Vega-Lite need manual positioning, anchoring, and updates: high annotation LOC, “Hard”
              intuitiveness and portability, and “High” error proneness.
      
          
      • 
          
    • 
            
      Primitives, weakly coupled
      
            
      d3-annotate, ggplot2, and ggplot2-annotate reduce LOC and initial effort, but weak semantic coupling
              still means manual adjustment when things change. HighCharts eases initial authoring with similar
              limits. Visual editors are easy to use but detach annotations from the data.
      
          
      • 
          
    • 
            
      Embedded in the grammar
      
            
      With annotations attached to data points, axes, and chart elements, updates to the visualization
              propagate to the annotations, which lowers error proneness and keeps them portable without
              reauthoring.
      
          
      • 
        
    
      
    
    
    
    
    
    
      
    
        
    
          
    Cite the paper
    
          
    
            Md Dilshadur Rahman, Md Rahat-uz-Zaman, Andrew McNutt, and Paul Rosen.
            AnnoGram: An Annotative Grammar of Graphics Extension.
            2025 IEEE Visualization and Visual Analytics (VIS), pp. 236–240, 2025.
            doi:10.1109/VIS60296.2025.00053
          
    
          
    
            Preprint on arXiv:2507.04236. Try the prototype in the
            playground.
            This work was supported in part by NSF award 2402719.
          
    
        
    
        
    
          
    
            BibTeX
            
          
    
          
    @inproceedings{rahman2025annogram,
  author    = {Rahman, Md Dilshadur and
               Rahat-uz-Zaman, Md and
               McNutt, Andrew and
               Rosen, Paul},
  title     = {{AnnoGram}: An Annotative
               Grammar of Graphics
               Extension},
  booktitle = {2025 IEEE Visualization
               and Visual Analytics (VIS)},
  pages     = {236--240},
  year      = {2025},
  doi       = {10.1109/VIS60296.2025.00053}
}