README
¶
Automated log and analytics collection for Dremio clusters
IMPORTANT LINKS
- Read the FAQ for common questions on setting up DDC
- Read the ddc.yaml for a full, detailed list of customizable collection parameters (optional)
- Read the official Dremio Support page for more details on the DDC architecture
Install DDC on your local machine
Download the latest release binary:
- Unzip the binary
- Open a terminal and change to the directory where you unzipped your binary
- Run the command
./ddc -h. If you see the DDC command help, you are good to go.
Dremio on Kubernetes
DDC connects via SSH or kubectl and collects a series of logs and files for Dremio, then puts those collected files in an archive
For Kubernetes deployments (Requires kubectl cluster access to be configured):
coordinator only
ddc --k8s --namespace mynamespace --coordinator app=dremio-coordinator
coordinator and executors
ddc --k8s --namespace mynamespace --coordinator app=dremio-coordinator --executors app=dremio-executor
to collect job profiles, system tables, kv reports and wlm (via REST API)
Requires Dremio admin privileges. Dremio PATs can be enabled by the support key auth.personal-access-tokens.enabled
ddc --k8s -n mynamespace -c app=dremio-coordinator -e app=dremio-executor --dremio-pat-prompt
Dremio on-prem
Specific executors that you want to collect from with the -e flag and coordinators with the -c flag. Specify SSH user, and SSH key to use.
For SSH based communication to VMs or Bare Metal hardware:
coordinator only
ddc --coordinator 10.0.0.19 --ssh-user myuser
coordinator and executors
ddc --coordinator 10.0.0.19 --executors 10.0.0.20,10.0.0.21,10.0.0.22 --ssh-user myuser
to collect job profiles, system tables, kv reports and wlm (via REST API)
Requires Dremio admin privileges. Dremio PATs can be enabled by the support key auth.personal-access-tokens.enabled
ddc --coordinator 10.0.0.19 --executors 10.0.0.20,10.0.0.21,10.0.0.22 --ssh-user myuser --dremio-pat-prompt
to avoid using the /tmp folder on nodes
ddc --coordinator 10.0.0.19 --executors 10.0.0.20,10.0.0.21,10.0.0.22 --ssh-user myuser --transfer-dir /mnt/lots_of_storage/
Dremio AWSE
Log-only collection from a Dremio AWSE coordinator is possible via the following command. This will produce a tarball with logs from all nodes.
./ddc awselogs
Dremio Cloud
To collect job profiles, system tables, and wlm via REST API, specify the following parameters in ddc.yaml
is-dremio-cloud: true
dremio-endpoint: "[eu.]dremio.cloud" # Specify whether EU Dremio Cloud or not
dremio-cloud-project-id: "<PROJECT_ID>"
dremio-pat-token: "<DREMIO_PAT>"
tmp-output-dir: /full/path/to/dir # Specify local target directory
and run ./ddc local-collect from your local machine
Windows Users
If you are running DDC from Windows, always run in a shell from the C: drive prompt.
This is because of a limitation of kubectl ( see https://github.com/kubernetes/kubernetes/issues/77310 )
ddc.yaml
The ddc.yaml file is located next to your DDC binary and can be edited to fit your environment. The default-ddc.yaml documents the full list of available parameters.
Flags
Available Commands:
awselogs Log only collect of AWSE from the coordinator node
completion Generate the autocompletion script for the specified shell
help Help about any command
local-collect retrieves all the dremio logs and diagnostics for the local node and saves the results in a compatible format for Dremio support
version Print the version number of DDC
Flags:
-c, --coordinator string coordinator to connect to for collection. With ssh set a list of ip addresses separated by commas. In K8s use a label that matches to the pod(s).
--coordinator-container string for use with -k8s flag: sets the container name to use to retrieve logs in the coordinators (default "dremio-master-coordinator,dremio-coordinator")
--ddc-yaml string location of ddc.yaml that will be transferred to remote nodes for collection configuration (default "/Users/ryan.svihla/Documents/GitHub/dremio-diagnostic-collector/bin/ddc.yaml")
-t, --dremio-pat-prompt Prompt for Dremio Personal Access Token (PAT)
-e, --executors string either a common separated list or a ip range of executors nodes to connect to. With ssh set a list of ip addresses separated by commas. In K8s use a label that matches to the pod(s).
--executors-container string for use with -k8s flag: sets the container name to use to retrieve logs in the executors (default "dremio-executor")
-h, --help help for ddc
-k, --k8s use kubernetes to retrieve the diagnostics instead of ssh, instead of hosts pass in labels to the --coordinator and --executors flags
-p, --kubectl-path string where to find kubectl (default "kubectl")
-n, --namespace string namespace to use for kubernetes pods (default "default")
--output-file string name of tgz file to save the diagnostic collection to (default "diag.tgz")
-s, --ssh-key string location of ssh key to use to login
-u, --ssh-user string user to use during ssh operations to login
-b, --sudo-user string if any diagnostics commands need a sudo user (i.e. for jcmd)
--transfer-dir string directory to use for communication between the local-collect command and this one (default "/tmp/ddc-20240112141658")
Documentation
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd package contains all the command line flag and initialization logic for commands
|
cmd package contains all the command line flag and initialization logic for commands |
|
local
cmd package contains all the command line flag and initialization logic for commands
|
cmd package contains all the command line flag and initialization logic for commands |
|
local/apicollect
apicollect provides all the methods that collect via the API, this is a substantial part of the activities of DDC so it gets it's own package
|
apicollect provides all the methods that collect via the API, this is a substantial part of the activities of DDC so it gets it's own package |
|
local/conf
package conf provides configuration for the local-collect command
|
package conf provides configuration for the local-collect command |
|
local/conf/autodetect
package autodetect looks at the system configuration and file names and tries to guess at the correct configuration
|
package autodetect looks at the system configuration and file names and tries to guess at the correct configuration |
|
local/consent
package consent contains the logic for showing what files are collected as well the boilerplate text
|
package consent contains the logic for showing what files are collected as well the boilerplate text |
|
local/ddcio
ddcio include helper code for io operations common to ddc
|
ddcio include helper code for io operations common to ddc |
|
local/jvmcollect
package jvmcollect handles parsing of the jvm information
|
package jvmcollect handles parsing of the jvm information |
|
local/logcollect
package logcollect contains the logic for log collection in the local-collect sub command
|
package logcollect contains the logic for log collection in the local-collect sub command |
|
local/nodeinfocollect
package nodeinfocollect has all the methods for collecting the information for nodeinfo
|
package nodeinfocollect has all the methods for collecting the information for nodeinfo |
|
local/queriesjson
queriesjson package contains the logic for collecting queries.json information
|
queriesjson package contains the logic for collecting queries.json information |
|
local/threading
threading package provides support for simple concurrency and threading
|
threading package provides support for simple concurrency and threading |
|
root/cli
package cli provides wrapper support for executing commands, this is so we can test the rest of the implementations quickly.
|
package cli provides wrapper support for executing commands, this is so we can test the rest of the implementations quickly. |
|
root/collection
collection package provides the interface for collection implementation and the actual collection execution
|
collection package provides the interface for collection implementation and the actual collection execution |
|
root/helpers
helpers package provides some general functions that do not have a good home
|
helpers package provides some general functions that do not have a good home |
|
root/kubernetes
kubernetes package provides access to log collections on k8s
|
kubernetes package provides access to log collections on k8s |
|
root/ssh
ssh package uses ssh and scp binaries to execute commands remotely and translate the results back to the calling node
|
ssh package uses ssh and scp binaries to execute commands remotely and translate the results back to the calling node |
|
version
cmd package contains all the command line flag and initialization logic for commands
|
cmd package contains all the command line flag and initialization logic for commands |
|
pkg
|
|
|
clusterstats
package clusterstats provides a placeholder for summary information found inside of a tarball, used by local and remote collect
|
package clusterstats provides a placeholder for summary information found inside of a tarball, used by local and remote collect |
|
consoleprint
package consoleprint contains the logic to update the console UI
|
package consoleprint contains the logic to update the console UI |
|
dirs
dirs provides helpers for working with directories on the filesystem
|
dirs provides helpers for working with directories on the filesystem |
|
jps
jps package provides logic for extracting values from jps
|
jps package provides logic for extracting values from jps |
|
masking
masking hides secrets in files and replaces them with redacted text
|
masking hides secrets in files and replaces them with redacted text |
|
output
output provides functinos around capturing output
|
output provides functinos around capturing output |
|
simplelog
simplelog package provides a simple logger
|
simplelog package provides a simple logger |
|
tests
package tests provides helper functions and mocks for running tests
|
package tests provides helper functions and mocks for running tests |
Click to show internal directories.
Click to hide internal directories.