Remote debugging nakama server go plugin with goland and delve

Hello there, I am new to nakama and go in gerenal and I am trying to get my local setup done, so I can start developing the custom server go plugin for the nakama server. I am stuck in the process of getting delve hooked up so I can debug the go plugin at runtime.

What I have done so far:

  1. Followed this tutorial to build the go plugin inside the docker image via a docker and docker-compose file because I am developing on Windows, and we can not build the plugin there :frowning:

  2. Modified the docker and docker compose file to hopefully
    a) run the postgres db in a separate docker container
    b) connect to a docker network
    c) use the nakama-dysm image
    d) add delve and
    e) execute nakama via delve command,
    by following various tutorials, how to get delve and goland remote debugging running.

The current docker file:

FROM heroiclabs/nakama-pluginbuilder:3.12.0 AS go-builder
WORKDIR /backend
COPY go.mod .
COPY *.go .
COPY vendor/ vendor/
RUN go build -gcflags "all=-N -l" --trimpath --mod=vendor --buildmode=plugin -o ./

FROM golang:1.14-alpine AS build
RUN apk add --update --no-cache git && rm -rf /var/cache/apk/*
COPY . .
RUN CGO_ENABLED=0 go get -ldflags "-s -w -extldflags '-static'"
FROM heroiclabs/nakama-dsym:3.12.0
COPY --from=build /go/bin/dlv /dlv
COPY --from=go-builder /backend/ /nakama/data/modules/
COPY local.yml /nakama/data/
CMD [ "/dlv" ]

The current docker compose file:

version: "3"
    build: .
      - SYS_PTRACE
      - seccomp:unconfined
      - "7349:7349"
      - "7350:7350"
      - "7351:7351"
      - "2306:2306"
    command: "--listen=:2306 --headless=true --log=true --log-output=debugger,debuglineerr,gdbwire,lldbout,rpc --accept-multiclient --api-version=2 exec /nakama/nakama -- --name nakama1 --database.address username:pw@postgres:5432/nakama --config /nakama/data/local.yml"
    name: nakamanetwork

If I start the docker via following goland configuration:

Nakama gets started correctly by delve, and I have the following output by delve:

API server listening at: [::]:2306
2022-06-01T18:57:32.784507737Z 2022-06-01T18:57:32Z warning layer=rpc Listening for remote connections (connections are not authenticated nor encrypted)
2022-06-01T18:57:32.784525788Z 2022-06-01T18:57:32Z debug layer=rpc API server pid = 1
2022-06-01T18:57:32.784528803Z 2022-06-01T18:57:32Z info layer=debugger launching process with args: [/nakama/nakama --name nakama1 --database.address nakamagameserver:root@postgres:5432/nakama --config /nakama/data/local.yml]
2022-06-01T18:57:33.171403892Z 2022-06-01T18:57:33Z warning layer=debugger reading debug_info: concrete subprogram without address range at 0x9daa77
2022-06-01T18:57:33.231283126Z 2022-06-01T18:57:33Z debug layer=rpc serving JSON-RPC on new connection
2022-06-01T18:57:33.231313980Z 2022-06-01T18:57:33Z debug layer=rpc (async 1) <- RPCServer.State(rpc2.StateIn{"NonBlocking":true})
2022-06-01T18:57:33.232185523Z 2022-06-01T18:57:33Z debug layer=rpc (async 1) -> rpc2.StateOut{"State":{"Pid":0,"Running":false,"Recording":false,"CoreDumping":false,"currentThread":{"id":12,"pc":140254288601232,"file":"","line":0,"goroutineID":0,"ReturnValues":null,"CallReturn":false},"Threads":[{"id":12,"pc":140254288601232,"file":"","line":0,"goroutineID":0,"ReturnValues":null,"CallReturn":false}],"NextInProgress":false,"WatchOutOfScope":[],"exited":false,"exitStatus":0,"When":""}} error: ""
2022-06-01T18:57:33.237038497Z 2022-06-01T18:57:33Z debug layer=rpc <- RPCServer.GetVersion(api.GetVersionIn{})
2022-06-01T18:57:33.237066327Z 2022-06-01T18:57:33Z debug layer=rpc -> *api.GetVersionOut{"DelveVersion":"Version: 1.8.3\nBuild: $Id: f92bb46b82b3b92d79ce59c4b55eeefbdd8d040c $","APIVersion":2,"Backend":"native","TargetGoVersion":"Go cmd/compile go1.18.2","MinSupportedVersionOfGo":"1.16.0","MaxSupportedVersionOfGo":"1.18.0"} error: ""
2022-06-01T18:57:33.238761651Z 2022-06-01T18:57:33Z debug layer=rpc <- RPCServer.Recorded(rpc2.RecordedIn{})
2022-06-01T18:57:33.238792770Z 2022-06-01T18:57:33Z debug layer=rpc -> *rpc2.RecordedOut{"Recorded":false,"TraceDirectory":""} error: ""
2022-06-01T18:57:33.240184265Z 2022-06-01T18:57:33Z debug layer=rpc <- RPCServer.IsMulticlient(rpc2.IsMulticlientIn{})
2022-06-01T18:57:33.240203125Z 2022-06-01T18:57:33Z debug layer=rpc -> *rpc2.IsMulticlientOut{"IsMulticlient":true} error: ""
2022-06-01T18:57:33.241380199Z 2022-06-01T18:57:33Z debug layer=rpc <- RPCServer.AttachedToExistingProcess(rpc2.AttachedToExistingProcessIn{})
2022-06-01T18:57:33.241396716Z 2022-06-01T18:57:33Z debug layer=rpc -> *rpc2.AttachedToExistingProcessOut{"Answer":false} error: ""
2022-06-01T18:57:33.242679231Z 2022-06-01T18:57:33Z debug layer=rpc <- RPCServer.FindLocation(rpc2.FindLocationIn{"Scope":{"GoroutineID":-1,"Frame":0,"DeferredCall":0},"Loc":"main.main","IncludeNonExecutableLines":false,"SubstitutePathRules":null})
2022-06-01T18:57:33.248352601Z 2022-06-01T18:57:33Z debug layer=rpc -> *rpc2.FindLocationOut{"Locations":[{"pc":33492498,"file":"","line":64,"function":{"name":"main.main","value":33492480,"type":0,"goType":0,"optimized":false},"pcs":[33492498]}]} error: ""
2022-06-01T18:57:33.250919881Z 2022-06-01T18:57:33Z debug layer=rpc <- RPCServer.ListSources(rpc2.ListSourcesIn{"Filter":""})
2022-06-01T18:57:33.251876139Z 2022-06-01T18:57:33Z debug layer=rpc -> *rpc2.ListSourcesOut{"Sources":["/_/os/user/cgo_lookup_unix.go","/_/os/user/getgrouplist_unix.go","/_/plugin/plugin_dlopen.go","/_/runtime/cgo/\u003cbuilt-in\u003e","/_/runtime/cgo/gcc_amd64.S","/_/runtime/cgo/gcc_context.c","/_/runtime/cgo/gcc_fatalf.c","MANY MANY MORE"]} #THIS LINE KEEPS GOING FOR ~ 100k Characters, NO JOKE
2022-06-01T18:57:33.258402593Z 2022-06-01T18:57:33Z debug layer=rpc <- RPCServer.CreateBreakpoint(rpc2.CreateBreakpointIn{"Breakpoint":{"id":0,"name":"","addr":0,"addrs":null,"file":"","line":17,"Cond":"","HitCond":"","continue":false,"traceReturn":false,"goroutine":false,"stacktrace":0,"LoadArgs":null,"LoadLocals":null,"WatchExpr":"","WatchType":0,"hitCount":null,"totalHitCount":0,"disabled":false}})
2022-06-01T18:57:33.258546991Z 2022-06-01T18:57:33Z debug layer=rpc -> *rpc2.CreateBreakpointOut{"Breakpoint":{"id":0,"name":"","addr":0,"addrs":null,"file":"","line":0,"Cond":"","HitCond":"","continue":false,"traceReturn":false,"goroutine":false,"stacktrace":0,"LoadArgs":null,"LoadLocals":null,"WatchExpr":"","WatchType":0,"hitCount":null,"totalHitCount":0,"disabled":false}} error: "could not find statement at, please use a line with a statement"
2022-06-01T18:57:33.260443590Z 2022-06-01T18:57:33Z debug layer=rpc (async 9) <- RPCServer.State(rpc2.StateIn{"NonBlocking":true})
2022-06-01T18:57:33.261049432Z 2022-06-01T18:57:33Z debug layer=rpc (async 9) -> rpc2.StateOut{"State":{"Pid":0,"Running":false,"Recording":false,"CoreDumping":false,"currentThread":{"id":12,"pc":140254288601232,"file":"","line":0,"goroutineID":0,"ReturnValues":null,"CallReturn":false},"Threads":[{"id":12,"pc":140254288601232,"file":"","line":0,"goroutineID":0,"ReturnValues":null,"CallReturn":false}],"NextInProgress":false,"WatchOutOfScope":[],"exited":false,"exitStatus":0,"When":""}} error: ""
2022-06-01T18:57:33.263408870Z 2022-06-01T18:57:33Z debug layer=rpc (async 10) <- RPCServer.Command(api.DebuggerCommand{"name":"continue","ReturnInfoLoadConfig":null})
2022-06-01T18:57:33.263426196Z 2022-06-01T18:57:33Z debug layer=debugger continuing
2022-06-01T18:57:33.288686524Z {"level":"info","ts":"2022-06-01T18:57:33.288Z","caller":"server/config.go:86","msg":"Successfully loaded config file","path":"/nakama/data/local.yml"}
2022-06-01T18:57:33.289005641Z {"level":"info","ts":"2022-06-01T18:57:33.288Z","caller":"v3/main.go:103","msg":"Nakama starting"}
2022-06-01T18:57:33.564896478Z {"level":"info","ts":"2022-06-01T18:57:33.564Z","caller":"v3/main.go:184","msg":"Startup done"}

But the breakpoints inside goland are showing erros and dont work. The main.go file of the plugin has a breakpoint that show the following error in goland:

A:\GameServerProject\main.go:17 could not find statement at, please use a line with a statement

And the breakpoints inside a different file of the go plugin that contains a test rpc shows this error in goland:

A:\GameServerProject\OwnRPC.go:16 Cannot find debugger path for -A:\GameServerProject/OwnRPC.go

For me, it looks like the first breakpoint is directing to nakamas main.go and the second breakpoints source file cannot be found at all on the remote.

What do I have messed up? Does anyone know what I am doing wrong? Do you have an idear? Or is there a detailed tutorial how to debug go plugins with delve that are getting build the way I described above? I found this statement that delve debugging should work, can anyone provide a docker file that is known good and I can have a look at? I feel like I’m missing something obvious…:exploding_head:

Thank you in advance!

Hi @dimisen,

can you check in Goland that you have Enable Go modules integration checked in Settings -> Go -> Go Modules?

Hi @ftkg, thanks for taking time!

I think you relate to this setting in goland? Yes i have checked this. I forogt to mention that, sorry. So sadly that isnt the problem :frowning:

Still no luck getting debugging running, even on Linux. I really want to get going with nakama, but that situation of not being able to debug with breakpoints holds me off admitting to it. How do you guys debug your go runtime modules, only with logs? Or am I just too stupid to get it running?

Hey @dimisen, we’ll return to you after testing it with Goland.
As an aside, you can try using VSCode, as that seems to work for some people.

1 Like

Hey! Any updates on using GoLand? I’ve been using the same and facing similar issues.

A temporary fix I’ve found is toggling the Go module setting in GoLand after launching the debugger. However, every time I relaunch the container, I’ve to toggle the option and re-run the debugger.

Hi @thealphadollar

We now have a guide for debugging Go server runtime code with Delve (Nakama: Debugging with Delve | Heroic Labs Documentation) however, there was a known issue with setting deferred breakpoints in IDEs that appears to have been fixed in Delve as of this PR (proc,terminal: allow setting suspended breakpoints by aarzilli · Pull Request #3154 · go-delve/delve · GitHub).

As soon as GoLand updates its internal version of Delve you should be able to debug using breakpoints. Unfortunately exactly when this update will occur in GoLand is out of our hands.

Thanks, @tom for the response. We’ve found a hack around the same using Docker - a similar setup provided in the guide mentioned by you.

I’ll leave it here in case anyone needs it in the future.