Dedupcsv
This utility removed duplicate rows. The input must be sorted!
Use -help to show:
$ dedupcsv -help
Help Message
Usage: dedupcsv [options]
NOTE: must be sorted; only compares row against prior row. -headers
CSV has headers (default true)
-help
Show help message
-i string
Input CSV filename; default STDIN
-keep
Keep CSV headers on output (default true)
-o string
Output CSV filename; default STDOUT
For example:
$ cat test1.csv
A,B,C
1,2,3
1,2,3
4,5,6
4,5,6
d,e,f
d,e,f
d,e,f
$ go run dedupcsv.go < test1.csv
A,B,C
1,2,3
4,5,6
d,e,f
$