We use nakama on a variety of platforms - developers use windows, we have a test environment running on a mac, and we will eventually run a production environment on linux droplets (until such time that we can afford your managed services).
I understand the recommendation for go module development is to use the docker images along with the docker build tool. But there’s some confusion about how to manage a production environment. Apparently the compiled shared objects for go modules are not transferable from platform to platform? And the recommendation for a production environment is NOT to use Docker, but to use binaries. So how does this then apply to go modules on the production environment - they need to be compiled there?
And for windows, does the docker build tool work, or is it still the case go can’t use the plugin buildmode on windows.
developers use windows, we have a test environment running on a mac, and we will eventually run a production environment on linux droplets
This sounds fine we’ve crafted the technology to minimize the amount of infrastructure required to run in various environments so it’s as easy as possible for local development too.
I understand the recommendation for go module development is to use the docker images along with the docker build tool.
This recommendation is in general because it means you don’t need to bother to configure and setup containers with Docker Compose the environment is composed of all the containers needed and are started together.
But there’s some confusion about how to manage a production environment. Apparently the compiled shared objects for go modules are not transferable from platform to platform?
No. Like any compiled language the shared object generated by the Go toolchain is architecture specific. When you use our pluginbuilder Docker image the compiled code on the host filesystem is built for the Linux x64 architecture. You can see this with:
$> file ./your-compiled-module.so
ELF 64-bit LSB shared object x86-64, version 1 (SYSV), dynamically linked
And the recommendation for a production environment is NOT to use Docker, but to use binaries.
This has come up a couple of times recently. There’s no production recommendation not to use Docker images or to suggest that native binaries are the right way. The only strong recommendation is NOT to use Docker Compose to run your server environment.
And for windows, does the docker build tool work, or is it still the case go can’t use the plugin buildmode on windows.
The Go language and compiler toolchain does not support plugins built on Windows. This is a constraint which comes from the Go team. It may change in the future but we’d recommend against Windows for your host OS with production instances anyway.
OK, so it sounds like if I use the docker build tool on a Linux environment, then move to the .so to a linux based VM for production, that should work?
And if compilation for go modules is platform specific, and can’t be build on windows, then go modules simply don’t work on windows using the windows binary?
This is a limitation of the current Go plugin system, it’s only available on Linux and macOS but we support Windows development too by using Docker to run Nakama and building plugins through our pluginbuilder.
one more clarification needed. Are go plugins that are built with the docker build tool, for the same platform, compatible with the nakama binaries? We have the latest linux binary running in our production environment, and compiled our go source code with go 1.13 to a shared object, and moved it to sever, and it doesn’t work.
Just to note, we have successfully build the same go source code, using the docker tool for mac os, running nakama from docker in our dev environment.
As a hail mary, I installed the go 1.13 tool kit on our production server, set GOPATH to our source code folder, did go module init, and go get the nakama common package 1.0.0. The shared object built, but still will not run in the server, we’re still getting the error:
plugin was built with a different version of package internal/cpu
This is using nakama 2.7.0 linux binary.
Hi, were you able to resolve this? Facing the same issue with v2.9.1. We just migrated up from 2.6.0 and thus this is a big change to new modules methodology of go1.13. Here is my go.mod, just to be sure that I am importing the right dependencies:
module reingames.com/pool
go 1.13
require (
github.com/google/uuid v1.1.1
github.com/heroiclabs/nakama-common v1.3.0
github.com/huandu/facebook v2.3.1+incompatible
github.com/maddevsio/fcm v1.0.4
reingames.com/common v0.0.0-00010101000000-000000000000 // This is a local package
)
replace reingames.com/common => ../../../../reingames // location of the local package
EDIT: I had left out -trimpath from the build command, which is recommended. I added the same, which fixed the problem., but now I have issue with a new package:
plugin was built with a different version of package golang.org/x/sys/unix
Here is my build command - in case it helps:
go build -buildmode=plugin -trimpath -o ./modules/pool.so
Looks like one of your dependencies doesn’t match the version of that same package Nakama is built with. There’s likely more than one mismatch but the runtime fails fast on the first one it finds.
Go is very particular on all package versions fully matching, so you should check Nakama’s modules.txt for your version of Nakama and make sure your module’s vendor/modules.txt contains the exact same versions. If it doesn’t you’ll need to ensure your module requires the correct versions - do not edit the modules.txt file manually, it’s generated by go mod commands.
You may find this post useful Dependency conflict errors when using Agones SDK in runtime module.
Zyro, thanks for the response, I had imagined so. I went through and cleared up all the other dependencies and only had the following:
module reingames.com/pool
go 1.13
require (
github.com/google/uuid v1.1.1
github.com/heroiclabs/nakama-common v1.3.0
reingames.com/common v0.0.0-00010101000000-000000000000
)
replace reingames.com/common => ../../reingames
Still got the same error. Then I looked at the link you shared and the solution suggested by Novabyte, edited the code to look as follows, bringing in all the dependencies of reingames package as dependencies here:
import (
"context"
"database/sql"
"encoding/json"
"errors"
"fmt"
"time"
"reingames.com/common/utilities"
"reingames.com/pool/authentication"
"reingames.com/pool/classes"
"reingames.com/pool/constants"
"reingames.com/pool/leagues"
"reingames.com/pool/matchloop"
"reingames.com/pool/nonmatchdata"
"reingames.com/pool/premiumclubs"
"reingames.com/pool/privateclubs"
_ "github.com/aws/aws-sdk-go" // dependency in reingames.com/common
"github.com/heroiclabs/nakama-common/api"
"github.com/heroiclabs/nakama-common/runtime"
_ "github.com/lib/pq" // dependency in reingames.com/common
_ "golang.org/x/crypto/ssh/terminal" // dependency in reingames.com/common
_ "golang.org/x/sys/unix" // added here to check if this resolves the issue, but it didn't
)
//go.mod as follows
module reingames.com/pool
go 1.13
require (
github.com/aws/aws-sdk-go v1.28.9
github.com/google/uuid v1.1.1
github.com/heroiclabs/nakama v2.5.1+incompatible
github.com/heroiclabs/nakama-common v1.3.0
github.com/lib/pq v1.3.0
golang.org/x/crypto v0.0.0-20200117160349-530e935923ad
golang.org/x/sys v0.0.0-20190412213103-97732733099d
reingames.com/common v0.0.0-00010101000000-000000000000
)
replace reingames.com/common => ../../reingames
Still the same error. Quick note: I found the exact same sequence of events in gitter chat with another user.
- Firstly the user had issues with package internal/cpu,
- novabyte suggested that he added
-trimpathto the build command - He did and then he ran into the issue with
golang.org/x/sys/unix
Unfortunately no solution is posted there either. But I have pinged him for help as well. Just in case this rings a bell with you?
BTW, I am building the nakama binary also from the source code (since I thought that would be the best way to avoid conflict in dependencies). What would be the full command with all options to build the nakama binary? Would it be:
env CGO_ENABLED=1 go build -trimpath
@gupta-vaibhav If you build from source you must make sure you check out the exact version of the official releases:
git checkout v2.9.1
env CGO_ENABLED=1 go build -trimpath -mod=vendor
I can see from your mod file that you’ve not updated your Go code correctly because it appears you reference the old “github.com/heroiclabs/nakama/runtime” import path which was used before the move to Go modules for dependency management. Please update your code fully to ensure that your mod file only shows the Nakama common dependency: “github.com/heroiclabs/nakama-common/runtime” etc.
When you’ve updated your code ensure that you run go mod tidy to cleanup the mod cache. If you continue to have problems run the command below and share the file here:
go mod graph > depsgraph.txt
@novabyte thanks for the suggestions. Here is what I have done again, just to be sure that everything is in order. To build the nakama binary:
vaibhavgupta@Vaibhavs-MacBook-Pro nakama % git checkout v2.9.1
HEAD is now at 8f527a71... Nakama 2.9.1 release.
vaibhavgupta@Vaibhavs-MacBook-Pro nakama % env CGO_ENABLED=1 go build -trimpath -mod=vendor
vaibhavgupta@Vaibhavs-MacBook-Pro nakama % ls -al nakama
-rwxr-xr-x 1 vaibhavgupta staff 53225516 Jan 29 06:17 nakama
vaibhavgupta@Vaibhavs-MacBook-Pro nakama % cp -i nakama /usr/local/bin
overwrite /usr/local/bin/nakama? (y/n [n]) y
vaibhavgupta@Vaibhavs-MacBook-Pro nakama %
I have also attached the log of how I run the server including a dump of go.mod before and after tidy and a dump of the output of the mod graph.
Many thanks for your help.
run server.txt (11.9 KB)
If you check the Nakama 2.9.1 source tree you’ll see both the go.mod file and vendor/modules.txt file specify the golang.org/x/sys dependency as v0.0.0-20190919044723-0c1ff786ef13.
Unless I’m misreading your output (which is possible, you’ve pasted a bunch of things) you’re specifying version v0.0.0-20190412213103-97732733099d. Worth noting that even if you don’t explicitly specify a dependency in your own go.mod file the toolchain will still resolve a version for it if it’s a transitive dependency, for example.
Make sure the version matches and your problem will go away.
@zyro thanks for your help. I did change the version of golang.org/x/sys dependency to match it as suggested by you. The error disappeared. But now the error is showing up for package github.com/aws/aws-sdk-go/aws/awserr
I have double checked and updated the version of aws-sdk-go to match with what is required by nakama. Here is my current go.mod:
module reingames.com/pool
go 1.13
require (
github.com/aws/aws-sdk-go v1.23.16
github.com/google/uuid v1.1.1
github.com/heroiclabs/nakama-common v1.3.0
github.com/lib/pq v1.3.0
github.com/stretchr/testify v1.4.0 // indirect
golang.org/x/crypto v0.0.0-20200117160349-530e935923ad
golang.org/x/sys v0.0.0-20190919044723-0c1ff786ef13
reingames.com/common v0.0.0-00010101000000-000000000000
)
replace reingames.com/common => ../../reingames
As you can see, the version of aws-sdk-go is v1.23.16, same as that in nakama’s go.mod. Still I am getting the same error:
{"level":"error","ts":"2020-01-30T09:34:31.197+0530","msg":"Could not open Go module","path":"modules/pool.so","error":"plugin.Open(\"modules/pool\"): plugin was built with a different version of package github.com/aws/aws-sdk-go/aws/awserr","stacktrace":"github.com/heroiclabs/nakama/v2/server.openGoModule\n\tgithub.com/heroiclabs/nakama/v2@/server/runtime_go.go:1973\ngithub.com/heroiclabs/nakama/v2/server.NewRuntimeProviderGo\n\tgithub.com/heroiclabs/nakama/v2@/server/runtime_go.go:1887\ngithub.com/heroiclabs/nakama/v2/server.NewRuntime\n\tgithub.com/heroiclabs/nakama/v2@/server/runtime.go:453\nmain.main\n\tgithub.com/heroiclabs/nakama/v2@/main.go:130\nruntime.main\n\truntime/proc.go:203"}
{"level":"error","ts":"2020-01-30T09:34:31.197+0530","msg":"Error initialising Go runtime provider","error":"plugin.Open(\"modules/pool\"): plugin was built with a different version of package github.com/aws/aws-sdk-go/aws/awserr","stacktrace":"github.com/heroiclabs/nakama/v2/server.NewRuntime\n\tgithub.com/heroiclabs/nakama/v2@/server/runtime.go:455\nmain.main\n\tgithub.com/heroiclabs/nakama/v2@/main.go:130\nruntime.main\n\truntime/proc.go:203"}
{"level":"fatal","ts":"2020-01-30T09:34:31.197+0530","msg":"Failed initializing runtime modules","error":"plugin.Open(\"modules/pool\"): plugin was built with a different version of package github.com/aws/aws-sdk-go/aws/awserr","stacktrace":"main.main\n\tgithub.com/heroiclabs/nakama/v2@/main.go:132\nruntime.main\n\truntime/proc.go:203"}
Am I doing something horribly wrong. Building with go v1.13.6 should make dependencies easier, ideally. Is there an alternate approach that I need to take to building which would eliminate these issues?
Not sure if this helps… I did the following: I commented the parts in my code that needed aws-sdk-go and tried to run the server. It was able to load the modules. Thus the only problem remaining is with aws sdk, that’s for sure.
Now, my game server code is accessing the aws files from the following location:
$GOPATH/pkg/mod/github.com/aws/aws-sdk-go@v1.23.16/
However nakama is using the aws sdk from its vendor folder, i.e.
$GOPATH/src/github.com/heroiclabs/nakama/vendor/github.com/aws/aws-sdk-go/
I have not found a way to check the version of the aws-sdk in nakama’s vendor folder but is it possible that it is a version different from 1.23.16?
Not sure if this makes sense?
EDIT: I found the version in nakama’s aws sdk folder, i.e., aws/version.go and the version is indeed v1.23.16
Nakama is definitely using v1.23.16 of github.com/aws/aws-sdk-go. Check that your plugin is built with that version too.
- Have you run
go mod vendorin your plugin? - Does your plugin’s
vendor/modules.txtshow version v1.23.16 forgithub.com/aws/aws-sdk-go? (Not thego.modfile, thevendor/modules.txtfile.) - Are you building your plugin with
go build -buildmode=plugin -trimpath -mod=vendor?
@zyro I really appreciate your patience and guidance in resolving this issue.
- Hi yes, I am using v1.23.16 as can be seen here, that’s the only version I have:
vaibhavgupta@Vaibhavs-MacBook-Pro go % ls ~/go/pkg/mod/github.com/aws
aws-sdk-go@v1.23.16
And my go.mod also specifies: github.com/aws/aws-sdk-go v1.23.16
- I ran
go mod vendorupon your suggestion and thevendor/modules.txtfile also confirms the same.
# github.com/aws/aws-sdk-go v1.23.16
github.com/aws/aws-sdk-go/aws
github.com/aws/aws-sdk-go/aws/awserr
...
- Yes I am building the plugin with the same command. Here is the exact command from my shell script I’ve written to ensure plugin is compiled each time before I restart the server:
if go build -buildmode=plugin -trimpath -mod=vendor -o ./modules/pool.so; then
nakama --config ./config/conf.yml --runtime.path ./modules/
else
echo "Compliation failed, check errors"
fi
What other information can I provide?
Regards,
EDIT: Even the aws sdk in my vendor folder is v1.23.16. Please see the following logs:
vaibhavgupta@Vaibhavs-MacBook-Pro aws % pwd
/Users/vaibhavgupta/goprojects/pool/go/vendor/github.com/aws/aws-sdk-go/aws
vaibhavgupta@Vaibhavs-MacBook-Pro aws % head version.go
// Package aws provides core functionality for making requests to AWS services.
package aws
// SDKName is the name of this AWS SDK
const SDKName = "aws-sdk-go"
// SDKVersion is the version of this SDK
const SDKVersion = "1.23.16"
Finally, I’ve got code to work, but with no additional insight on what to change. Here’s what I did.
- Moved nakama code out of $GOPATH (not sure if I were supposed to do it in the first place)
- Edited nakama’s go.mod to forcefully point to (using replace) my local aws-sdk-go@v1.23.16
- Added a log at the beginning of nakama’s main.go so that I could be sure that I am running the freshly complied binary and not somehow an old version
- Edited my plugin’s go.mod to forcibly point to (using replace) my local aws-sdk-go@v1.23.16
Guess what - still the same error of
plugin was built with a different version of github.com/aws/aws-sdk-go
However, I gave up trying it on my Mac and tried to do everything fresh on our AWS instance where we are running our dev server. Guess what! It ran in the first attempt, without any issue at all.
I am still clueless as to why it isn’t working locally on my Mac. I guess, I will just clean up everything here, do fresh installations of go and get fresh pull of the nakama as well as my server code and will give it a shot again.
For now, just wanted to update that the server booted up perfectly fine on the dev server. Thanks for all the patience and guidance. On the brighter side, the whole experience taught me a lot about how modules / dependencies are handled in the new module approach of go.
Best,
@gupta-vaibhav You mention at the top that you’ve got it to work but most of the post continues to describe that it doesn’t work. Did you solve the setup issues to get the codebase to build and load at runtime?
As a side note you should never mix GOPATH and module-based builds with the Go toolchain. In fact since Go 1.13 you should consider the GOPATH approach deprecated. It was one of the reasons we had to move Nakama over to use module-based builds in the first place.