It all started with the need to measure 4G internet bandwidth at strategic locations across India. I quickly wrote a bash script to run iperf3 client against an iperf3 server hosted by us. The bandwidth measurement results are uploaded to another server in the cloud. A systemd timer periodically runs the script.

Everything worked as expected in the beginning. But, soon we started seeing many occurrences of the error iperf3: error - the server is busy running a test. try again later all over our results. My investigation reveals that by design, iperf3 server doesn’t allow simultaneous tests from more than one client: the actual reason unknown, maybe historic design decision?

The conventional solution to such problem is to run iperf3 server on multiple machines/containers and expose them through a layer 4 load balancer like AWS NLB. But I didn’t choose that approach due to the additional cost and complexity. Instead, I chose to use a simple TCP load balancer called Balance.

The setup

The plan is simple, run multiple instances/processes of iperf3 server in the same host but on different ports. Then we run Balance specifying the load balancing port and the target hosts. In our case, the hostname of our target hosts are localhost as both iperf3 servers and the load balancer are running on the same machine. So only the difference in the target hosts is the port. The command-lines for the setup are below.

1
2
3
4
5
6
7
8
$ iperf3 -s -p 2101 &
$ iperf3 -s -p 2102 &
$ iperf3 -s -p 2103 &
$ iperf3 -s -p 2104 &
$ iperf3 -s -p 2105 &
$ balance -f 2100 localhost:2101 localhost:2102 localhost:2103 localhost:2104 localhost:2105

Once the setup is ready, I tried testing with iperf3 client running on my laptop. The client connected, but it didn’t progress with the testing. The client simply hung. I could also witness the client connection in the servers. But strangely, two servers reported the same client connection. Bit of a web search, I found this iperf3 issue. Seems iperf3 uses two connections for its operation: one for control, one for data. In the load balancing setup, each of this connection ended up in different iperf3 servers as they balanced on a round-robin basis. This explains why the two iperf3 servers reported the same client connection and why the test didn’t progress.

I know AWS load balancers allow us to configure sticky sessions for such use case. Balance doesn’t seem to provide similar functionality. But it does support hash-based balancing which uses hashed client address to determine the target host. Though it is not same as a sticky session, it satisfies our need for using the same target host for all connections of a client. The command-line option to enable hash balancing is %. So the revised command-line looks like below.

1
2
...
$ balance -f 2100 localhost:2101 localhost:2102 localhost:2103 localhost:2104 localhost:2105 %

Putting it together

Though the above command-lines do the job, they don’t provide the flexibility to run/stop them quickly. So I crafted the following bash script to run them with a single command and stop them with Ctrl+C key press. I hope it would be useful for someone.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
#!/bin/bash

trap "kill 0" EXIT

lb_port=5100
port_start=5101
port_end=5105
hosts=""

for (( port=$port_start; port<=$port_end; port++ ))
do
    echo "start iperf3 server on port $port..."
    iperf3 -s -p $port &
    
    hosts="$hosts localhost:$port"
done

Links

1. iperf3 project page
2. Balance project page
3. Balance man page

I started experiencing this problem when I was adding a new API. Our distribution platform archives movie content after N days. I was implementing an API that lists movies from the archives along with information like the size of each movie and the time taken for retrieving every movie from the archives.

Model:

This is the struct definition we use to list movies.

1
2
3
4
5
6
7
8
9
type Response struct {
  Movies []Movie
}

type Movie struct {
  ID                    string
  AvailabilityInMinutes int    `json:"availabilityInMinutes,omitempty"`
  Size                  uint64 `json:"size,omitempty"`
}

Let us consider that this API returns 2 movies of which the first movie is available immediately while the other movie is not present in the archives.

Data Initialisation

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
movies := []Movie{
  {
    ID: "3f00c091-b29d-4b99-8136-5cb65c987250", 
    AvailabilityInMinutes: 0,
    Size: 54626741
  },
  {
    ID: "04ea246b-7e7a-46a4-ba56-867117a90610", 
  }
}

moviesJSON, err := json.Marshal(movies)
if err != nil {
  handleErr(err)
}

Ideally, for the first movie, the API should list all the properties. The API should only list the id property for the second movie since the system doesn’t have any information about the movie.

Expected Response:

1
2
3
4
5
6
7
8
9
10
11
12
{
  "movies": [
    {
      "id": "3f00c091-b29d-4b99-8136-5cb65c987250",
      "availabilityInMinutes": "0",
      "size": "54626741"
    },
    {
      "id": "04ea246b-7e7a-46a4-ba56-867117a90610"
    }
  ]
}

The Problem

However, the response that we got was:

1
2
3
4
5
6
7
8
9
10
11
{
  "movies": [
    {
      "id": "3f00c091-b29d-4b99-8136-5cb65c987250",
      "size": "54626741"
    },
    {
      "id": "04ea246b-7e7a-46a4-ba56-867117a90610"
    }
  ]
}

The availability field for both movies is missing! For the first movie,availabilityInMinutes has been omitted since it holds 0 which is the default value/empty value for int datatype. For the second movie,availabilityInMinutes has been omitted since the attribute does not hold any value. If we remove the omitempty tag in that field, then the second movie that is not present in the system will also have availabilityInMinutes set to 0 which is incorrect.

After discussing with the team, we decided to use the pointer type.

The solution:

The fix is to use the pointer to the primitive type as the pointers have nil as the zero values.

Updated Model

1
2
3
4
5
type Movie struct {
  ID                    string
  AvailabilityInMinutes *int   `json:"availabilityInMinutes,omitempty"`
  Size                  uint64 `json:"size,omitempty"`
}

Data

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
movie1Availability := 371367181
movies := []Movie{
  {
    ID: "3f00c091-b29d-4b99-8136-5cb65c987250",
    AvailabilityInMinutes: new(int),
    Size: 54626741
  },
  {
    ID: "04ea246b-7e7a-46a4-ba56-867117a90610",
  }
}

moviesJSON, err := json.Marshal(movies)
if err != nil {
  handleErr(err)
}

On marshaling the above data, we get the following JSON:

1
2
3
4
5
6
7
8
9
10
11
12
{
  "movies": [
    {
      "id": "3f00c091-b29d-4b99-8136-5cb65c987250",
      "availabilityInMinutes": 0,
      "size": "54626741"
    },
    {
      "id": "04ea246b-7e7a-46a4-ba56-867117a90610"
    }
  ]
}

And the problem was solved!

TL;DR

Using the pointers (whose zero values are nil) in place of the primitive types solves the problem.

The Go programming language has libraries for creation and migration of SQLite databases. This post covers the basics of SQLite DB creation and migration using Go, and provides guidelines on how to update the DB schema without loss or corruption of data. This is based on one of our applications, where we encountered this problem of schema migration.

Note: If you are aware of how to create and setup migration for databases using Go, please skip the Let Us GO! section below.

Migration Scripts And Versioning

A good way of shipping out a client side database application is to equip it with the mechanism to create the DB itself, when launched for the first time. This can be done by embedding a set of scripts containing SQL commands to do the same. We refer to these files as migration scripts. The naming of these files is done in a specific format: <version>_<description>.<up/down>.sql. This conveys the order in which these must be executed.

• The version is the numerical value used to determine the schema version of the database.
• The description denotes the changes made by the script. This is ignored in the migration process.
• The up/down suffix determines whether the script will be used to upgrade or downgrade the database.

If the database for the application does not exist (as would be the case for the first launch), the migration code shall run the up scripts to create it. The schema version of the database shall be set to the latest version specified among the scripts. For subsequent launches, the migration code searches for any scripts that have a higher version, and executes the SQL statements in those scripts. For instance, if the DB schema version is 6, only the scripts with version 7 or higher, if available, shall be used to update the schema.

Let Us GO!

This post shall utilize the following Go libraries and packages for creation and migration of the DB:

• database/sql: Generic interface around SQL databases.
• go-sqlite3: Database driver for SQLite. This shall be used in conjunction with database/sql.
• golang-migrate: Used for database migrations.
• go-bindata: Used for embedding the migration scripts into the application.

The process to equip migration into the application is threefold:

1. Convert the migration scripts into embeddable binary form
2. Create the database in the application
3. Run the relevant migration scripts on the database

A. Using go-bindata

The following command shall generate the binary data:

1
go-bindata -o <path-to-datafile>.go -prefix "<migration-scripts-dir>" -pkg <package-name> <migration-scripts-dir>

This shall convert the scripts into binary form and store this information in the specifed Go file. The -pkg specifies the package for the generated Go file. The path to this file, and the migration scripts directory, should be in the directory of the specified package.

B. Creating The Database

The database/sql package, along with the go-sqlite3 driver, can be used to create the DB. The following code snippet performs this task:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
import (
	"database/sql"

	_ "github.com/mattn/go-sqlite3"
	"github.com/pkg/errors"
)

func NewDB(dbPath string) (*sql.DB, error) {
	sqliteDb, err := sql.Open("sqlite3", dbPath)
	if err != nil {
		return nil, errors.Wrap(err, "failed to open sqlite DB")
	}

	return sqliteDb, nil
}

C. Running the Migration Scripts

The generated Go data file contains the migration scripts listed as Assets. We shall use this in the migration function, as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
import (
	"fmt"

	"github.com/golang-migrate/migrate"
	"github.com/golang-migrate/migrate/database/sqlite3"
	bindata "github.com/golang-migrate/migrate/source/go_bindata"
)

func RunMigrateScripts(db *sql.DB) error {
	driver, err := sqlite3.WithInstance(db, &sqlite3.Config{})
	if err != nil {
		return fmt.Errorf("creating sqlite3 db driver failed %s", err)
	}

	res := bindata.Resource(AssetNames(),
		func(name string) ([]byte, error) {
			return Asset(name)
		})

	d, err := bindata.WithInstance(res)
	m, err := migrate.NewWithInstance("go-bindata", d, "sqlite3", driver)
	if err != nil {
		return fmt.Errorf("initializing db migration failed %s", err)
	}

	err = m.Up()
	if err != nil && err != migrate.ErrNoChange {
		return fmt.Errorf("migrating database failed %s", err)
	}

	return nil
}

The m.Up() call executes the up scripts, whichever necessary. For downgrading the DB, this can be replaced by m.Down().

D. Putting It All Together

Assuming that we have a db package for the DB related code, the final product looks something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
package db

import (
	"database/sql"
	"fmt"

	"github.com/golang-migrate/migrate"
	"github.com/golang-migrate/migrate/database/sqlite3"
	bindata "github.com/golang-migrate/migrate/source/go_bindata"
	_ "github.com/mattn/go-sqlite3"
	"github.com/pkg/errors"
)

func NewDB(dbPath string) (*sql.DB, error) {
	sqliteDb, err := sql.Open("sqlite3", dbPath)
	if err != nil {
		return nil, errors.Wrap(err, "failed to open sqlite DB")
	}

	return sqliteDb, nil
}

func RunMigrateScripts(db *sql.DB) error {
	driver, err := sqlite3.WithInstance(db, &sqlite3.Config{})
	if err != nil {
		return fmt.Errorf("creating sqlite3 db driver failed %s", err)
	}

	res := bindata.Resource(AssetNames(),
		func(name string) ([]byte, error) {
			return Asset(name)
		})

	d, err := bindata.WithInstance(res)
	m, err := migrate.NewWithInstance("go-bindata", d, "sqlite3", driver)
	if err != nil {
		return fmt.Errorf("initializing db migration failed %s", err)
	}

	err = m.Up()
	if err != nil && err != migrate.ErrNoChange {
		return fmt.Errorf("migrating database failed %s", err)
	}

	return nil
}

The final step is using these two in the main application.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
package main

import (
	"log"

	"<db-package-import-path>"
	_ "github.com/mattn/go-sqlite3"
)

func main() {
	sqliteDb, err := db.NewDB("<path-to-db-file>")
	if err != nil {
		log.Fatal(err)
	}

	defer sqliteDb.Close()

	err = db.RunMigrateScripts(sqliteDb.DB)
	if err != nil {
		log.Fatal(err)
	}

	log.Info("successfully migrated DB..")

	// rest of application main
}

Guidelines for SQLite DB Schema Migration Scripts

As mentioned earlier, SQLite has a slight disadvantage compared to other database systems. On most DB systems, the ALTER TABLE statement can be used quite effectively to change the table structure in the database. However, for SQLite databases, this statement can only perform two operations:

• Change the name of a table
• Add columns to a table

Other operations, such as renaming or removing a column, or changing the data type of a column, cannot be accomplished with the ALTER TABLE statement. For these operations, a four-step process needs to be followed:

1. Rename the existing table.
2. Create the new table with the same name that the table originally had.
3. Populate this newly created table with information from the old table.
4. Drop the old table.

The following is a template for the the up migration scripts following this process:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
ALTER TABLE <table_name> RENAME TO _<table_name>_old;

CREATE TABLE <table_name>
(
	column1 datatype [NULL | NOT NULL],
	column1 datatype [NULL | NOT NULL],
	...
);

INSERT INTO <table_name> (column1, column2, ...)
	SELECT column1, column2, ...
	FROM _<table_name>_old;

DROP TABLE _<table_name>_old

The corresponding down migration scripts shall do the inverse.

Conclusion

This post provides a detailed process for writing a client side Go application with a database, without requiring to install any DB system on the host machine. It also provides a safe, albeit slightly indirect, way of migrating the database schema without worrying about any loss or malformation of data on it.

Imagine for a moment that you have a microservice written in the Go Programming Language that is deployed on more than one instance for reliabilty and performance reasons. They all share the same underlying PostgreSQL database. Perhaps you then want to limit certain functionality to only one instance at a time (e.g. background worker, queue consumption). To achieve this synchronization, you decide to use PostgreSQL session-level advisory locks. You initialize a session-level advisory lock, pass it to the concerned functionality and then wrap it with a lock and unlock on the session-level advisory lock.

So far, things sound good. Only one instance enters the critical section. All other instances are blocked from entering the critical section till the first instance leaves it. However, it turns out that once the first instance unlocks the session-level advisory lock and leaves the critical section, no instance (including the first instance) can enter the critical section again. They all block when acquiring the advisory lock.

Sometimes, the critical section can be entered for a second time, and sometimes even for three or more times. But finally the end result is the same. Sooner or later, all instances block on acquiring the session-level advisory lock and none can enter the synchronized portion of code forever.

The offending code…

This is the situation we found ourselves in. We went through our code. Locking and unlocking a pg_advisory_lock was done like below:

Locking

1
2
db, _ := sql.Open("postgres", _<url>_)
db.Exec(`SELECT pg_advisory_lock($1)`, <a session id>)

Unlocking

1
db.Exec(`SELECT pg_advisory_unlock($1)`, <the same session id>)

Finding the culprit!

After consulting our in-house expert and digging a little deeper, we discovered a couple of facts:

• Session-level pg_advisory_locks can only be released in the same database connection in which it was obtained. For more details, see here.
• Go’s standard library sql package creates a pool of database connections by default. See here. Each DB call is done on an arbitrary connection from the DB pool.

So deducing from the above, the DB connection pool must not be returning the same connection for the unlock that was used for the lock. So how do we enforce that the lock and the unlock are done on the same connection? It turns out that a single connection can be obtained from and released to the DB pool.

The right solution.

The proper way to lock and unlock session-level pg_advisory_locks is to first obtain a connection from the DB connection pool and store it and use it for both operations.

Locking

1
2
3
db, _ := sql.Open("postgres", _<url>_)
conn, _err_ := db.Conn(context.Background())
conn.ExecContext(context.Background(), `SELECT pg_advisory_lock($1)`, <a session id>)

Unlocking

1
2
conn.ExecContext(context.Background(), `SELECT pg_advisory_unlock($1)`, <the same session id>)
conn.Close()

The size of the DB connection pool should be increased appropriately to compensate for the connections that are permanently taken away to be used in this way.

Conclusion

Moral of the story: Always unlock a PostgreSQL session-level advisory lock on the same connection which was used to lock it, otherwise sooner or later, you will end up not unlocking it, and then you will be blocked forever. Fortunately, this kind of problem will show up in your face immediately rather than later. Unfortunately, the solution is not straightforward, but a little subtle. Hopefully, instead of rolling back the changes and avoiding or limiting the use of PostgreSQL session-level advisory locks, this post will help you solve the problem correctly.

If you are on the rush of finding the solution, scroll down all the way to the end.

We were developing a feature, in which, we had to configure the host machine IP address on a remote machine so that the remote machine can pull data at its ease. As the host machine is connected to multiple networks, the challenge is to identify the right network interface that is accessible by the remote machine and configure the corresponding IP address on the remote machine. If we fail to pass on the right network interface to the receiver, our whole purpose of building that feature will be collapsed. So it turned out to be an unprecedented requirement for us to solve.

For the purpose of illustration, let us assume that we have two networks connected to the machine and one of the two networks is a public network and the other is a private network.

Given this setup, our goal was to find the network interface the packet takes to reach its destination. A bit of schooling - If a packet wants to reach a host located in a private network, will take Network Interface 1 as its exit route, likewise a packet takes Network Interface 2 to reach a host in public network.

Our first thought solution was to leverage the Routing table. Fortuitously, we found the Linux command in no time to query the routing table to extract the desired information (network interface). By formatting the command with the destination host IP address we will be able to figure out the network interface a packet takes to reach the destination host. The ip route command looks like the command below.

1
ip route get 172.217.160.132

That seems to be an easy-peasy solution to the problem! Isn’t it? Integrating the raw ip route command programmatically just as the snippet below solves the problem.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
// cmd = "ip route get 172.217.160.132"
func exe_cmd(cmd string, wg *sync.WaitGroup) {
	parts := strings.Fields(cmd)
	head := parts[0]
	parts = parts[1:len(parts)]

	out, err := exec.Command(head, parts...).Output()
	if err != nil {
		fmt.Printf("%s", err)
	}
	
	fmt.Printf("%s", out)
	wg.Done()
}

The output will look similar to the below output.

1
172.217.160.132 via 172.17.0.1 dev eth1  src 172.17.0.2

The caveat with this approach is that you have to parse the output string to extract out the necessary information. Unfortunately, this process will turn out to be more painful in case of any error like network going down or not connected to the internet. Even if it’s plausible for us to parse for any possible output, it will eventually fail to meet the coding standards leaving the reviewers’ eyebrows frowned. We realized that we got to look for a neat and impeccable solution.

Further exploring led us to a very convincing solution in all the ways. The sub-package routing of gopacket gave us everything that we felt wanting in the previous approach. It prevented our code to give a room for string parsing logic and escaped us from damaging the readability of the code. As an added bonus, we enjoyed the ease of handling any errors on using routing package.

Without anything stopping us further, we jumped on to using routing sub-package leaving the Linux Command approach as an ephemeral hero.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
func determineRouteInterface(serverAddr string) error {
	var ip net.IP
	if ip = net.ParseIP(serverAddr); ip == nil {
		return fmt.Errorf("error as non-ip target %s is passed", serverAddr)
	}

	router, err := routing.New()
	if err != nil {
		return errors.Wrap(err, "error while creating routing object")
	}

	_, gatewayIP, preferredSrc, err := router.Route(ip)
	if err != nil {
		return errors.Wrapf(err, "error routing to ip: %s", serverAddr)
	}

	fmt.Printf("gatewayIP: %v preferredSrc: %v", gatewayIP, preferredSrc)
	return nil
}

The output of the above code snippet will look similar to the below output.

1
gatewayIP: 172.17.0.1 preferredSrc: 172.17.0.2

With no surprise, the output is very neatly present eliminating the need for fancy string parsing logic.

In the first place, we were totally surprised to see a package available in Golang to query the routing table on a Linux machine. We understood that the problem of determining a network interface a packet takes to reach the destination is very rare and least underscored on the web. Hence, we decided to bring the routing package to the light. Indeed, this blog post was a result of our excitement on finding a package to fetch routing information from an underlying routing table on a Linux machine ;)

It all began when our operation team complaint that a particular report is taking really lots of time to generate. Initial analysis revealed the report spent most of the time in executing a particular query in our Postgres database. Though the query is joining(inner) three tables, the join and where clause are straightforward. We also confirmed indexes exist for the columns involved in join and where clause. So we fired up pgHero and generated the execution plan for the query. Loading the execution plan into PEV indicated that the where clause on timestamp columns are not using the indexes and had to scan all rows.

To understand the problem deeper let us consider the following pseudo table models.

theatres table

screens table

licenses table

The report is about listing all theatre, screen pairs having any valid movie license within the specified date/time range. The offending query was written like below:

1
2
3
4
5
SELECT DISTINCT t.id AS theatre_id, s.id AS screen_id
FROM theatres t
INNER JOIN licenses l ON l.theatre_id = t.id
INNER JOIN screens s ON s.id = l.screen_id
WHERE (l.valid_from, l.valid_till) OVERLAPS ('2019-04-01T00:00:00+0000', '2019-05-31T23:59:59+0000')

The above query took ~450ms to select 140 rows on tables theatres, screens and licenses with row count of 37339, 147850, 350693 respectively. The execution plan indicated the query uses indexes for the various id columns but didn’t use the indexes for valid_from and valid_till column.

Suspecting the timestamp indexes, we rewrote the query like below to check the usage of operator OVERLAPS prevents the query using the indexes.

1
2
3
4
5
6
7
SELECT DISTINCT t.id AS theatre_id, s.id AS screen_id
FROM theatres t
INNER JOIN licenses l ON l.theatre_id = t.id
INNER JOIN screens s ON s.id = l.screen_id
WHERE (l.valid_from >= '2019-04-01T00:00:00+0000' AND l.valid_from <= '2019-05-31T23:59:59+0000')
    OR (l.valid_till >= '2019-04-01T00:00:00+0000' AND l.valid_till <= '2019-05-31T23:59:59+0000')
     OR (l.valid_from <= '2019-04-01T00:00:00+0000' AND l.valid_till >= '2019-05-31T23:59:59+0000')

To our surprise the query is fast, it took just ~180ms* to select the same **140 rows from the same dataset. The execution plan showed it uses the timestamp indexes. We couldn’t understand why the modified query is able to use indexes and faster, but the original query with OVERLAPS operator couldn’t and slower. We did a little search on the Internet but we couldn’t gather much info. But thinking about it aloud we realized rationale. The operator OVERLAPS checks whether two timestamp ranges overlaps or not. But the indexes are created for the individual timestamp columns. It is understandable that the operator can’t take advantage of the indexes as it requires a single timestamp range comprising both the start and end timestamps, i.e. the valid_from and valid_till. So the query had to scan all rows to construct the timestamp range, combined value of valid_from and valid_till, to check against the specified timestamp range.

As we realized the separate indexes doesn’t help time window overlap condition, we were set to find out is there a way to create an index for time window/timestamp ranges. We found that Postgres has builtin timestamp range data types tsrange (without time zone info) and tstzrange (with time zone info). So we replaced the two columns valid_from and valid_till with single column validity of type tstzrange. We also created a compatible index, GiST, for the column data. While migrating the data from existing columns, we also found invalid data of kind valid_till being less than valid_from. The tstzrange type’s builtin validation caught these invalid data and we were able to fix them.

licenses table with column validity

We rewrote the query like below to achieve the result with the new validity column of type tstzrange.

1
2
3
4
5
SELECT DISTINCT t.id AS theatre_id, s.id AS screen_id
FROM theatres t
INNER JOIN licenses l ON l.theatre_id = t.id
INNER JOIN screens s ON s.id = l.screen_id
WHERE l.validity && tstzrange('["2019-04-01T00:00:00+0000", "2019-05-31T23:59:59+0000"]')

Now the query is as fast as the previous modified version, took ~180ms to produce same 140 rows from same the dataset. But the usage of validity column with tstzrange type’s operator && made the query succinct. Additionally tstzrange type also brought in automatic validation for the timestamp range data.

We were happy that we not only made the query fast but also learned something new in Postgres. We hope this experience and learning of ours will be useful to you.

References

Postgres Range Types