Confusion around compiling go modules for different platforms

Hi @oscargoldman

developers use windows, we have a test environment running on a mac, and we will eventually run a production environment on linux droplets

This sounds fine we’ve crafted the technology to minimize the amount of infrastructure required to run in various environments so it’s as easy as possible for local development too.

I understand the recommendation for go module development is to use the docker images along with the docker build tool.

This recommendation is in general because it means you don’t need to bother to configure and setup containers with Docker Compose the environment is composed of all the containers needed and are started together.

But there’s some confusion about how to manage a production environment. Apparently the compiled shared objects for go modules are not transferable from platform to platform?

No. Like any compiled language the shared object generated by the Go toolchain is architecture specific. When you use our pluginbuilder Docker image the compiled code on the host filesystem is built for the Linux x64 architecture. You can see this with:

$> file ./your-compiled-module.so
ELF 64-bit LSB shared object x86-64, version 1 (SYSV), dynamically linked

And the recommendation for a production environment is NOT to use Docker, but to use binaries.

This has come up a couple of times recently. There’s no production recommendation not to use Docker images or to suggest that native binaries are the right way. The only strong recommendation is NOT to use Docker Compose to run your server environment.

And for windows, does the docker build tool work, or is it still the case go can’t use the plugin buildmode on windows.

The Go language and compiler toolchain does not support plugins built on Windows. This is a constraint which comes from the Go team. It may change in the future but we’d recommend against Windows for your host OS with production instances anyway.

OK, so it sounds like if I use the docker build tool on a Linux environment, then move to the .so to a linux based VM for production, that should work?

And if compilation for go modules is platform specific, and can’t be build on windows, then go modules simply don’t work on windows using the windows binary?

This is a limitation of the current Go plugin system, it’s only available on Linux and macOS but we support Windows development too by using Docker to run Nakama and building plugins through our pluginbuilder.

one more clarification needed. Are go plugins that are built with the docker build tool, for the same platform, compatible with the nakama binaries? We have the latest linux binary running in our production environment, and compiled our go source code with go 1.13 to a shared object, and moved it to sever, and it doesn’t work.

Just to note, we have successfully build the same go source code, using the docker tool for mac os, running nakama from docker in our dev environment.

As a hail mary, I installed the go 1.13 tool kit on our production server, set GOPATH to our source code folder, did go module init, and go get the nakama common package 1.0.0. The shared object built, but still will not run in the server, we’re still getting the error:
plugin was built with a different version of package internal/cpu
This is using nakama 2.7.0 linux binary.

Hi, were you able to resolve this? Facing the same issue with v2.9.1. We just migrated up from 2.6.0 and thus this is a big change to new modules methodology of go1.13. Here is my go.mod, just to be sure that I am importing the right dependencies:

module reingames.com/pool

go 1.13

require (
	github.com/google/uuid v1.1.1
	github.com/heroiclabs/nakama-common v1.3.0
	github.com/huandu/facebook v2.3.1+incompatible
	github.com/maddevsio/fcm v1.0.4
	reingames.com/common v0.0.0-00010101000000-000000000000 // This is a local package
)

replace reingames.com/common => ../../../../reingames // location of the local package

EDIT: I had left out -trimpath from the build command, which is recommended. I added the same, which fixed the problem., but now I have issue with a new package:

plugin was built with a different version of package golang.org/x/sys/unix

Here is my build command - in case it helps:

go build -buildmode=plugin -trimpath -o ./modules/pool.so

Looks like one of your dependencies doesn’t match the version of that same package Nakama is built with. There’s likely more than one mismatch but the runtime fails fast on the first one it finds.

Go is very particular on all package versions fully matching, so you should check Nakama’s modules.txt for your version of Nakama and make sure your module’s vendor/modules.txt contains the exact same versions. If it doesn’t you’ll need to ensure your module requires the correct versions - do not edit the modules.txt file manually, it’s generated by go mod commands.

You may find this post useful Dependency conflict errors when using Agones SDK in runtime module.

Zyro, thanks for the response, I had imagined so. I went through and cleared up all the other dependencies and only had the following:

module reingames.com/pool

go 1.13

require (
	github.com/google/uuid v1.1.1
	github.com/heroiclabs/nakama-common v1.3.0
	reingames.com/common v0.0.0-00010101000000-000000000000
)

replace reingames.com/common => ../../reingames

Still got the same error. Then I looked at the link you shared and the solution suggested by Novabyte, edited the code to look as follows, bringing in all the dependencies of reingames package as dependencies here:

import (
	"context"
	"database/sql"
	"encoding/json"
	"errors"
	"fmt"
	"time"

	"reingames.com/common/utilities"

	"reingames.com/pool/authentication"
	"reingames.com/pool/classes"
	"reingames.com/pool/constants"
	"reingames.com/pool/leagues"
	"reingames.com/pool/matchloop"
	"reingames.com/pool/nonmatchdata"
	"reingames.com/pool/premiumclubs"
	"reingames.com/pool/privateclubs"

	_ "github.com/aws/aws-sdk-go" // dependency in reingames.com/common
	"github.com/heroiclabs/nakama-common/api"
	"github.com/heroiclabs/nakama-common/runtime"
	_ "github.com/lib/pq" // dependency in reingames.com/common
	_ "golang.org/x/crypto/ssh/terminal" // dependency in reingames.com/common
	_ "golang.org/x/sys/unix" // added here to check if this resolves the issue, but it didn't
)

//go.mod as follows
module reingames.com/pool

go 1.13

require (
	github.com/aws/aws-sdk-go v1.28.9
	github.com/google/uuid v1.1.1
	github.com/heroiclabs/nakama v2.5.1+incompatible
	github.com/heroiclabs/nakama-common v1.3.0
	github.com/lib/pq v1.3.0
	golang.org/x/crypto v0.0.0-20200117160349-530e935923ad
	golang.org/x/sys v0.0.0-20190412213103-97732733099d
	reingames.com/common v0.0.0-00010101000000-000000000000
)

replace reingames.com/common => ../../reingames

Still the same error. Quick note: I found the exact same sequence of events in gitter chat with another user.

  • Firstly the user had issues with package internal/cpu,
  • novabyte suggested that he added -trimpath to the build command
  • He did and then he ran into the issue with golang.org/x/sys/unix

Unfortunately no solution is posted there either. But I have pinged him for help as well. Just in case this rings a bell with you?

BTW, I am building the nakama binary also from the source code (since I thought that would be the best way to avoid conflict in dependencies). What would be the full command with all options to build the nakama binary? Would it be:

env CGO_ENABLED=1 go build -trimpath

@gupta-vaibhav If you build from source you must make sure you check out the exact version of the official releases:

git checkout v2.9.1
env CGO_ENABLED=1 go build -trimpath -mod=vendor

I can see from your mod file that you’ve not updated your Go code correctly because it appears you reference the old “github.com/heroiclabs/nakama/runtime” import path which was used before the move to Go modules for dependency management. Please update your code fully to ensure that your mod file only shows the Nakama common dependency: “github.com/heroiclabs/nakama-common/runtime” etc.

When you’ve updated your code ensure that you run go mod tidy to cleanup the mod cache. If you continue to have problems run the command below and share the file here:

go mod graph > depsgraph.txt

@novabyte thanks for the suggestions. Here is what I have done again, just to be sure that everything is in order. To build the nakama binary:

vaibhavgupta@Vaibhavs-MacBook-Pro nakama % git checkout v2.9.1
HEAD is now at 8f527a71... Nakama 2.9.1 release.
vaibhavgupta@Vaibhavs-MacBook-Pro nakama % env CGO_ENABLED=1 go build -trimpath -mod=vendor
vaibhavgupta@Vaibhavs-MacBook-Pro nakama % ls -al nakama
-rwxr-xr-x  1 vaibhavgupta  staff  53225516 Jan 29 06:17 nakama
vaibhavgupta@Vaibhavs-MacBook-Pro nakama % cp -i nakama /usr/local/bin
overwrite /usr/local/bin/nakama? (y/n [n]) y
vaibhavgupta@Vaibhavs-MacBook-Pro nakama %

I have also attached the log of how I run the server including a dump of go.mod before and after tidy and a dump of the output of the mod graph.

Many thanks for your help.
run server.txt (11.9 KB)

1 Like

If you check the Nakama 2.9.1 source tree you’ll see both the go.mod file and vendor/modules.txt file specify the golang.org/x/sys dependency as v0.0.0-20190919044723-0c1ff786ef13.

Unless I’m misreading your output (which is possible, you’ve pasted a bunch of things) you’re specifying version v0.0.0-20190412213103-97732733099d. Worth noting that even if you don’t explicitly specify a dependency in your own go.mod file the toolchain will still resolve a version for it if it’s a transitive dependency, for example.

Make sure the version matches and your problem will go away.

@zyro thanks for your help. I did change the version of golang.org/x/sys dependency to match it as suggested by you. The error disappeared. But now the error is showing up for package github.com/aws/aws-sdk-go/aws/awserr

I have double checked and updated the version of aws-sdk-go to match with what is required by nakama. Here is my current go.mod:

module reingames.com/pool

go 1.13

require (
	github.com/aws/aws-sdk-go v1.23.16
	github.com/google/uuid v1.1.1
	github.com/heroiclabs/nakama-common v1.3.0
	github.com/lib/pq v1.3.0
	github.com/stretchr/testify v1.4.0 // indirect
	golang.org/x/crypto v0.0.0-20200117160349-530e935923ad
	golang.org/x/sys v0.0.0-20190919044723-0c1ff786ef13
	reingames.com/common v0.0.0-00010101000000-000000000000
)

replace reingames.com/common => ../../reingames

As you can see, the version of aws-sdk-go is v1.23.16, same as that in nakama’s go.mod. Still I am getting the same error:

{"level":"error","ts":"2020-01-30T09:34:31.197+0530","msg":"Could not open Go module","path":"modules/pool.so","error":"plugin.Open(\"modules/pool\"): plugin was built with a different version of package github.com/aws/aws-sdk-go/aws/awserr","stacktrace":"github.com/heroiclabs/nakama/v2/server.openGoModule\n\tgithub.com/heroiclabs/nakama/v2@/server/runtime_go.go:1973\ngithub.com/heroiclabs/nakama/v2/server.NewRuntimeProviderGo\n\tgithub.com/heroiclabs/nakama/v2@/server/runtime_go.go:1887\ngithub.com/heroiclabs/nakama/v2/server.NewRuntime\n\tgithub.com/heroiclabs/nakama/v2@/server/runtime.go:453\nmain.main\n\tgithub.com/heroiclabs/nakama/v2@/main.go:130\nruntime.main\n\truntime/proc.go:203"}
{"level":"error","ts":"2020-01-30T09:34:31.197+0530","msg":"Error initialising Go runtime provider","error":"plugin.Open(\"modules/pool\"): plugin was built with a different version of package github.com/aws/aws-sdk-go/aws/awserr","stacktrace":"github.com/heroiclabs/nakama/v2/server.NewRuntime\n\tgithub.com/heroiclabs/nakama/v2@/server/runtime.go:455\nmain.main\n\tgithub.com/heroiclabs/nakama/v2@/main.go:130\nruntime.main\n\truntime/proc.go:203"}
{"level":"fatal","ts":"2020-01-30T09:34:31.197+0530","msg":"Failed initializing runtime modules","error":"plugin.Open(\"modules/pool\"): plugin was built with a different version of package github.com/aws/aws-sdk-go/aws/awserr","stacktrace":"main.main\n\tgithub.com/heroiclabs/nakama/v2@/main.go:132\nruntime.main\n\truntime/proc.go:203"}

Am I doing something horribly wrong. Building with go v1.13.6 should make dependencies easier, ideally. Is there an alternate approach that I need to take to building which would eliminate these issues?

Dear @zyro @novabyte,

Not sure if this helps… I did the following: I commented the parts in my code that needed aws-sdk-go and tried to run the server. It was able to load the modules. Thus the only problem remaining is with aws sdk, that’s for sure.

Now, my game server code is accessing the aws files from the following location:
$GOPATH/pkg/mod/github.com/aws/aws-sdk-go@v1.23.16/

However nakama is using the aws sdk from its vendor folder, i.e.
$GOPATH/src/github.com/heroiclabs/nakama/vendor/github.com/aws/aws-sdk-go/

I have not found a way to check the version of the aws-sdk in nakama’s vendor folder but is it possible that it is a version different from 1.23.16?

Not sure if this makes sense?

EDIT: I found the version in nakama’s aws sdk folder, i.e., aws/version.go and the version is indeed v1.23.16

Nakama is definitely using v1.23.16 of github.com/aws/aws-sdk-go. Check that your plugin is built with that version too.

  • Have you run go mod vendor in your plugin?
  • Does your plugin’s vendor/modules.txt show version v1.23.16 for github.com/aws/aws-sdk-go? (Not the go.mod file, the vendor/modules.txt file.)
  • Are you building your plugin with go build -buildmode=plugin -trimpath -mod=vendor?

@zyro I really appreciate your patience and guidance in resolving this issue.

  1. Hi yes, I am using v1.23.16 as can be seen here, that’s the only version I have:
vaibhavgupta@Vaibhavs-MacBook-Pro go % ls ~/go/pkg/mod/github.com/aws
aws-sdk-go@v1.23.16

And my go.mod also specifies: github.com/aws/aws-sdk-go v1.23.16

  1. I ran go mod vendor upon your suggestion and the vendor/modules.txt file also confirms the same.
# github.com/aws/aws-sdk-go v1.23.16
github.com/aws/aws-sdk-go/aws
github.com/aws/aws-sdk-go/aws/awserr
...
  1. Yes I am building the plugin with the same command. Here is the exact command from my shell script I’ve written to ensure plugin is compiled each time before I restart the server:
if go build -buildmode=plugin -trimpath -mod=vendor -o ./modules/pool.so; then
        nakama --config ./config/conf.yml --runtime.path ./modules/
else
        echo "Compliation failed, check errors"
fi

What other information can I provide?

Regards,


EDIT: Even the aws sdk in my vendor folder is v1.23.16. Please see the following logs:

vaibhavgupta@Vaibhavs-MacBook-Pro aws % pwd
/Users/vaibhavgupta/goprojects/pool/go/vendor/github.com/aws/aws-sdk-go/aws
vaibhavgupta@Vaibhavs-MacBook-Pro aws % head version.go
// Package aws provides core functionality for making requests to AWS services.
package aws

// SDKName is the name of this AWS SDK
const SDKName = "aws-sdk-go"

// SDKVersion is the version of this SDK
const SDKVersion = "1.23.16"

Finally, I’ve got code to work, but with no additional insight on what to change. Here’s what I did.

  1. Moved nakama code out of $GOPATH (not sure if I were supposed to do it in the first place)
  2. Edited nakama’s go.mod to forcefully point to (using replace) my local aws-sdk-go@v1.23.16
  3. Added a log at the beginning of nakama’s main.go so that I could be sure that I am running the freshly complied binary and not somehow an old version
  4. Edited my plugin’s go.mod to forcibly point to (using replace) my local aws-sdk-go@v1.23.16

Guess what - still the same error of
plugin was built with a different version of github.com/aws/aws-sdk-go

However, I gave up trying it on my Mac and tried to do everything fresh on our AWS instance where we are running our dev server. Guess what! It ran in the first attempt, without any issue at all.

I am still clueless as to why it isn’t working locally on my Mac. I guess, I will just clean up everything here, do fresh installations of go and get fresh pull of the nakama as well as my server code and will give it a shot again.

For now, just wanted to update that the server booted up perfectly fine on the dev server. Thanks for all the patience and guidance. On the brighter side, the whole experience taught me a lot about how modules / dependencies are handled in the new module approach of go.

Best,

@gupta-vaibhav You mention at the top that you’ve got it to work but most of the post continues to describe that it doesn’t work. Did you solve the setup issues to get the codebase to build and load at runtime?

As a side note you should never mix GOPATH and module-based builds with the Go toolchain. In fact since Go 1.13 you should consider the GOPATH approach deprecated. It was one of the reasons we had to move Nakama over to use module-based builds in the first place.

It’s still not working locally on my mac, but it worked on our Linux server on AWS. For Mac, I will (later) try to remove all the files and make a fresh attempt to see if it does. It must be some random link to a different version that I am not able to find and break.

Here is a detailed step-by-step of how I migrated from 2.6.0 to a 2.7.0+ version on my Linux server, in case it helps anyone else trying to do the same. @novabyte @zyro please correct me / add. Relevant only for another layman who builds from the source, like I do.

  1. Make a goprojects directory outside $GOPATH
$ mkdir $HOME/goprojects
  1. Clone nakama repo in this new folder and build it there
$ cd $HOME/goprojects
$ git clone https://github.com/heroiclabs/nakama.git
$ cd nakama
$ git checkout v2.9.1
$ env CGO_ENABLED=1 go build -trimpath -mod=vendor

3 Copy / move / clone / (create new) project directory inside this folder

$ cp $GOPATH/src/pool $HOME/goprojects/
  1. Initialise a go module in this folder
$ cd $HOME/goprojects/pool
$ go mod init "reingames.com/pool"
  1. Update the dependencies in the project to import nakama-common instead of nakama, e.g.
// In the go code, remove the following from your import list
"github.com/heroiclabs/nakama/runtime"
// replace it with
"github.com/heroiclabs/nakama-common/runtime"
  1. Link any local packages that you may have with their physical local locations, e.g. this is how I linked my local reingames.com/common dependency
//In my plugin's go.mod
require (
	// all other dependencies here
	reingames.com/common v0.0.0
)
// The following line will link reingames.com/common package to it's local physical location 
replace reingames.com/common => ../../reingames
  1. Make sure that everything is set up nice and clean. And build
$ go mod tidy
$ go mod vendor
  1. Build and run the code
$ go build -buildmode=plugin -trimpath -mod=vendor -o ./modules/pool.so
$ $HOME/nakama/nakama --config ./config/conf.yml

If you still get plugin was made with a different version of package x/y/z here are a few tips that I got and tried from @novabyte and @zyro

  1. Check the version of dependency in the modules.txt file in the vendor folder (e.g. $HOME/gopath/pool/vendor/modules.txt)
  2. If the version you find here is different from the one that is in nakama’s folder, you could do the following:
  • If the package x/y/z is imported directly in your source code, the package will be included in your plugin’s go.mod file. Set the version there to match the version that in nakama’s
  • If the package x/y/z is not imported directly and not showing up in your plugin’s go.mod, force import it and set the version, i.e.
// In the import part of your main.go
    _ "x.y.z" // import this. Not the leading _ (underscore)

// In go.mod file
    x.y.z v1.2.3 // match this version with the one in nakama's go.mod

I don’t know if it’s just basic stuff that is obvious to all, but still posting in the hope that this helps someone else as well.

2 Likes