Troubleshoot problems

When a machine is not working as expected, follow this debugging workflow: check logs, enable debug logging if needed, access the machine remotely, and if necessary use advanced diagnostics.

For common errors and their solutions, see Common Errors.

Check logs

Machine shows as offline

If your machine shows as offline in the Viam app, you need to check logs on the machine itself.

First determine whether viam-server was installed with viam-agent (most common) or manually:

ps aux | grep viam-agent
root      566431  0.5  0.2 1247148 20336 ?       Ssl  11:24   0:00 /opt/viam/bin/viam-agent --config /etc/viam.json

If you see a viam-agent process, you used viam-agent to install.

sudo journalctl --unit=viam-agent

sudo journalctl --unit=viam-server

Machine shows as online

Check for errors on the CONFIGURE page

Go to your machine’s CONFIGURE page and look for a red exclamation icon on any resource card. Click the icon or expand the ERROR LOGS panel to see resource-specific errors.

Check logs on the LOGS tab

Go to the LOGS tab and look for errors.

You can filter logs by:

• Level: Error, Warn, Info, Debug
• Keyword: full text search (supports regular expressions)
• Time range: set start and end times, or use live mode to stream logs in real time
• Resource: filter by a specific component or service name

The default log level is Info. If you are not seeing helpful logs, enable debug logging.

Enable debug logging

For all resources on the machine, add "debug": true to the JSON configuration:

{
  "debug": true,
  "components": [{ ... }]
}

For individual resources, click the … menu on the resource’s configuration card and select Enable debug logs.

For resources matching a pattern, configure the log field in the machine configuration:

"log": [
  {
    "pattern": "rdk.components.arm",
    "level": "debug"
  },
  {
    "pattern": "rdk.services.*",
    "level": "debug"
  }
]

For more on log configuration, see Logging.

By default, viam-server deduplicates log messages that repeat within a one-minute window. To disable this, see Disable log deduplication.

Access logs from the command line

viam machines logs --machine <machine-name-or-id> --levels error,warn

Stream logs in real time:

viam machines part logs --part <part-name-or-id> --tail

Filter by keyword and time range:

viam machines logs --machine <machine-name-or-id> --keyword "motor" --start 2026-04-07T10:00:00Z --end 2026-04-07T11:00:00Z

Remote shell access

Access a terminal on the machine without setting up SSH tunnels:

1. Add the ViamShellDanger fragment to your machine.

2. Open a shell:

   viam machines part shell --part <part-name-or-id>
   

3. Copy files from the machine:

   viam machines part cp --part <part-name-or-id> -r machine:/path/to/files ~/Downloads/
   

4. Tunnel to a specific port on the machine:

   viam machines part tunnel --part <part-name-or-id> --local-port 8080 --destination-port 8080
   

Debug endpoints

For performance issues, enable pprof profiling endpoints:

{
  "components": [],
  "services": [],
  "enable_web_profile": true
}

This exposes endpoints for CPU profiling, memory allocation analysis, goroutine blocking analysis, and execution tracing.

The /debug/graph endpoint renders an SVG visualization of the machine’s resource dependency graph.

For details, see Debug Endpoints.

Diagnostics

FTDC

FTDC (full-time diagnostic capture) continuously records machine metrics in a compressed binary format: resource counts, WebRTC connection stats, data manager upload stats, and more.

Download FTDC data from a machine:

viam machines part get-ftdc --part <part-name-or-id> ./diagnostics/

Parse and analyze FTDC data locally:

viam parse-ftdc --path ./diagnostics/ftdc-data

Traces

If tracing is enabled in your machine configuration, you can download and analyze OpenTelemetry traces:

viam traces get-remote --part <part-name-or-id> ./traces/

Print traces to the console:

viam traces print-remote --part <part-name-or-id>

Import traces into an OTLP-compatible tool (such as Jaeger):

viam traces import-remote --part <part-name-or-id> --endpoint localhost:4317

Restart a machine

1. Navigate to your machine’s page.

2. Click the part status dropdown to the right of your machine’s name.

3. If your machine was installed with viam-agent, click Restart. Both viam-server and viam-agent will restart.

   If you do not see a Restart button, click the … menu on the machine part card and select Restart part.

It takes a few minutes for viam-server to shut down and restart.

From the command line:

viam machines part restart --part <part-name-or-id>

Revert to an earlier configuration

Viam keeps a record of your configuration changes. To see the history, click History on the CONFIGURE tab.

To restore an earlier version, click the Restore version button next to the desired configuration.

Advanced debugging for Go modules

If a Go module hangs or does not shut down properly, send a SIGQUIT signal to get a stack trace:

kill -3 <PID>

The process dumps a stack trace to the viam-server logs showing all goroutines and where execution is blocked.

Get support

If you cannot resolve the issue, reach out on the Community Discord. Post the error message along with what you were doing when it occurred.

If asked, you can share your location with the Viam Support team by navigating to your location and clicking Add Viam support. Remove access after receiving support by clicking Remove Viam support.