README
¶
glazed - Output structured data in a variety of formats
Add the icing to your structured data!
Glazed is a library that makes it easy to output structured data. When programming, we have a rich understanding of the data we are working with, yet when we output it to the user, we are forced to use a flat, unstructured format.
It tries to implement some of the ideas listed in 14 great tips to make amazing command line applications.
It is in early alpha, and will change. Contributions are welcome, but this project is going to be an experimental playground for a while, while I try to figure out what is possible and worth tackling.
Features
With glazed, you can output object and table data in a rich variety of ways:
- as human-readable tables
+ glaze json misc/test-data/1.json misc/test-data/2.json misc/test-data/3.json
+-----+-----+------------+-----+-----+
| a | b | c | d.e | d.f |
+-----+-----+------------+-----+-----+
| 1 | 2 | [3 4 5] | 6 | 7 |
| 10 | 20 | [30 40 50] | 60 | 70 |
| 100 | 200 | [300] | | |
+-----+-----+------------+-----+-----+
- as CSV/TSV
+ glaze json misc/test-data/1.json misc/test-data/2.json misc/test-data/3.json --output csv
a,b,c,d.e,d.f
1,2,[3 4 5],6,7
10,20,[30 40 50],60,70
100,200,[300],,
- as markdown
+ glaze json misc/test-data/1.json misc/test-data/2.json misc/test-data/3.json --table-format markdown
+ glow -
A │ B │ C │ D E │ D F
──────┼─────┼────────────┼─────┼──────
1 │ 2 │ [3 4 5] │ 6 │ 7
10 │ 20 │ [30 40 50] │ 60 │ 70
100 │ 200 │ [300] │ │
- as JSON
+ glaze json misc/test-data/2.json --output json
[
{
"a": 10,
"b": 20,
"c": [
30,
40,
50
],
"d": {
"e": 60,
"f": 70
}
}
]
You can flatten fields (happens by default when outputting to a table)
+ glaze json misc/test-data/2.json --output json --flatten
[
{
"a": 10,
"b": 20,
"c": [
30,
40,
50
],
"d.e": 60,
"d.f": 70
}
- as YAML
+ glaze json --output yaml ./misc/test-data/1.json
- a: 1
b: 2
c:
- 3
- 4
- 5
d:
e: 6
f: 7
You can select and reorder fields:
+ glaze json misc/test-data/1.json misc/test-data/2.json misc/test-data/3.json --fields c,b,a --table-format markdown
+ glow -
C │ B │ A
─────────────┼─────┼──────
[3 4 5] │ 2 │ 1
[30 40 50] │ 20 │ 10
[300] │ 200 │ 100
You can filter out fields:
+ glaze json misc/test-data/1.json misc/test-data/2.json misc/test-data/3.json --filter d.e
+-----+-----+------------+-----+
| a | b | c | d.f |
+-----+-----+------------+-----+
| 1 | 2 | [3 4 5] | 7 |
| 10 | 20 | [30 40 50] | 70 |
| 100 | 200 | [300] | |
+-----+-----+------------+-----+
You can rename columns, using both simple string replacement or more advanced regexp matching:
+ glaze yaml misc/test-data/test.yaml --input-is-array --rename baz:blop,d.e:dang
+-----+------+-----+------+-----+---------+
| bar | blop | d.f | dang | foo | foobar |
+-----+------+-----+------+-----+---------+
| 7 | 2 | 7 | 6 | 1 | [3 4 5] |
| 70 | 20 | 70 | 60 | 10 | |
| | 200 | | | | [300] |
+-----+------+-----+------+-----+---------+
+ glaze yaml misc/test-data/test.yaml --input-is-array \
--rename-regexp '^(.*)bar:${1}blop','b..:blip'
+------+------+-----+-----+-----+---------+
| blip | blop | d.e | d.f | foo | fooblop |
+------+------+-----+-----+-----+---------+
| 2 | 7 | 6 | 7 | 1 | [3 4 5] |
| 20 | 70 | 60 | 70 | 10 | |
| 200 | | | | | [300] |
+------+------+-----+-----+-----+---------+
- use go templates to customize output
You can use go templates to either create a new field (called _0 per default). Per default, the templates are applied at the input level, when rows are actually still full blown objects (if reading in from JSON for example).
❯ glaze json misc/test-data/[123].json --template '{{.a}}-{{.b}}: {{.d.f}}'
+---------------------+
| _0 |
+---------------------+
| 1-2: 7 |
| 10-20: 70 |
| 100-200: <no value> |
+---------------------+
You can also apply templates at the row level, once the input has been flattened.
In this case, because flattened columns contain the symbol ., fields get renamed
to use the symbol _ as a separator.
❯ glaze json misc/test-data/[123].json --template '{{.a}}-{{.b}}: {{.d_f}}' \
--use-row-templates --fields a,_0 \
--output csv
a,_0
1,1-2: 7
10,10-20: 70
100,100-200: <no value>
Instead of just adding / replacing everything with a single field _0, you
can also specify multiple templates using the --template-field argument, which has
the form COLNAME:TEMPLATE.
❯ glaze json misc/test-data/[123].json \
--template-field 'foo:{{.a}}-{{.b}},bar:{{.d_f}}' \
--use-row-templates --fields a,foo,bar
+-----+---------+------------+
| a | foo | bar |
+-----+---------+------------+
| 1 | 1-2 | 7 |
| 10 | 10-20 | 70 |
| 100 | 100-200 | <no value> |
+-----+---------+------------+
To make things a bit more readable, especially when doing a lot of template transformations,
you can also load field templates from a yaml file using the @ symbol.
❯ glaze json misc/test-data/[123].json \
--template-field '@misc/template-field-object.yaml' \
--output json
[
{
"barbaz": "6 - 7",
"foobar": "1"
},
{
"barbaz": "60 - 70",
"foobar": "10"
},
{
"barbaz": "\u003cno value\u003e - \u003cno value\u003e",
"foobar": "100"
}
]
- output individual objects or rows as separate files
Glazed provides a variety of "middlewares" with which you can:
- flatten nested objects into rows
- create new fields based on go templates
- filter and reorder columns
of := formatters.NewCSVOutputFormatter()
of.AddTableMiddleware(middlewares.NewFlattenObjectMiddleware())
of.AddTableMiddleware(middlewares.NewFieldsFilterMiddleware(
[]string{"a", "b"},
[]string{"c"}
)
for _, obj := range objects {
of.AddRow(&types.SimpleRow{Hash: obj})
}
s, err := of.Output()
fmt.Println(s)
For easy integration into your own tools, glazed provides:
- a simple API for:
- input processors
- row and object middlewares
- output formatters
- bindings and helpers for:
- go command-line flag parsing
- cobra and viper libraries
- YAML driven configuration
Glazed also comes with the glaze tool which can be use for simple data manipulation and rich terminal output, leveraging the glazed library.
Getting started
Using the glaze command line tool
First, install the glaze tool.
- Show 4-5 cool examples
Developing with glazed
Write a tiny command line tool:
- if CLI flags can be set up quickly, do that
- generate a random table
- output it using glazed
Examples
Output formats
- json [x]
- yaml - #19
- csv [x]
- ascii [x]
- markdown [x]
- html [x]
File output
- Single file output - #4
- Multi file output - #4
Flattening structures
- json to rows
Filtering columns
- filters and fields
Go template support
- single string template
- multi file templates
- field templates
Markdown output and templating
- Multi markdown template output with index page
- markdown template file
Configuration file
- some examples
Schema documentation
- show how to output schema
Using glaze as a library
Middlewares
- ObjectMiddleware
- RowMiddleware
- TableMiddleware
Formatters
- TableOutputFormatter
- CsvOutputFormatter
Command Line Integration
- cobra integration [x]
- golang flags - #3
- add support for configuring command line flags (enable / disable / rename)
- viper integration
- calibrate from config files - #17
Schema documentation
- show how to load different schemas
The glaze tool
Installation
Run the glaze CLI by using go run ./cmd/glaze.
Import formats
- json / json rows / multiple files - #13
- yaml / multiple files - #14
- csv - #15
- cut / ascii - #16
- sqlite / SQL - #20
- binary parser
Output flags
Current RFCs
I keep a list of the current planned features as RFC documents.
- 01 - Flag helpers
- 02 - Multi-file output
- 03 - SQLite output
- 04 - Configuration files
- 05 - Glaze CLI tool
General brainstorm
- documentation for each subsystem
Future ideas
Glaze CLI
UX
- table app that can hide/show/rename/reorder columns
- markdown rendering with glow - #21
- style aliases (like pretty=oneline for git)
- maybe styles can also have additional parameters
- sparklines and other shenanigans
File Formats
- add support for arbitrary input / output SQL
- add support for inputting binary data and providing a parser
- add support for pcap input
- add support for excel input
- parquet format (and pandas? numpy?)
- excel export
- annotate excel export with as much metadata as possible
Transformation
- add jq support
- search engine / autocompletion based on known schema
- use query language to create hyperlinks in output
Misc glaze features
- add support for pushing to cloud resources
- dynamodb
- S3
- SQL connectors (see arbitrary input / output SQL)
- add support for serving over HTTP
- API server to render local data
- HTML frontend
- serve a local SQL database? meh...
- useful if you want the user to modify the DB? why not just output sqlite
- cloud / network API output forms, for example to store something in s3 or other databases
- SQL
- dynamodb
- s3
- do we want some kind of transformation DSL / configuration DSL to do more complicated things? Definitely not at first, before having the use case for it.
- hyperlinked schema definitions
- collect metadata and event logs to what led to the creation of the data itself
Directories