Nine reasons to switch from Python to Go

Published in

DataDrivenInvestor

9 min readFeb 25, 2020

Nine reasons to switch from Python to Go

Switching to a new programming language is often a big deal, especially when team members have a lot of experience with the original language. Earlier this year, Stream’s switched it’s main programming language from Python to Go. This article will explain some of the reasons why they decided to switch from Python to Go.

Reasons to use Go

Reason 1 — Performance

Golang and python performance comparison

Go is very fast. Its performance is close to that of Java or C. Go is 30 times faster than Python.

Reason 2 — The performance of the language itself is important

For many applications, the programming language is just the glue between the application and the database. The performance of the language itself is usually not important.

Stream is an API provider that provides feedback infrastructure to 500 companies and more than 200 million end users. For years, we’ve been optimizing the performance of software such as Cassandra, PostgreSQL, Redis, and so on, but now we’ve reached the limits of the programming language we’re using.

A Programmer's Guide to Creating an Eclectic Bookshelf

Every developer should have a bookshelf. The possible set of texts in his cabinet are myriad, but not every collection…

www.datadriveninvestor.com

Python is a great language, but for examples such as serialization/deserialization, sorting, and aggregation, it performs poorly. We often experience performance issues, Cassandra takes 1ms of time to retrieve data, and Python converts it into an object with the next 10ms.

Reason 3 — Developer efficiency, and not being too innovative

Take a look at the Go code from this section of the “How to Start Learning Go”tutorial.

type openWeatherMap struct{}func (w openWeatherMap) temperature(city string) (float64, error) {
    resp, err := http.Get("http://api.openweathermap.org/data/2.5/weather?APPID=YOUR_API_KEY&q=" + city)
    if err != nil {
        return 0, err
    }    defer resp.Body.Close()    var d struct {
        Main struct {
            Kelvin float64 `json:"temp"`
        } `json:"main"`
    }    if err := json.NewDecoder(resp.Body).Decode(&d); err != nil {
        return 0, err
    }    log.Printf("openWeatherMap: %s: %.2f", city, d.Main.Kelvin)
    return d.Main.Kelvin, nil
}

If you’re new to Go, there won’t be many surprises reading this code. It demonstrates assignments, data structures, pointers, formatting, and built-in HTTP libraries.

When I first came into contact with programming, I always liked to use Python’s advanced features. Python lets you get great ideas from the code you’re writing. For example, you can:

Register the class yourself with MetaClasses when the code is initialized
Switch True and False
Add a function to the list of built-in functions
Overload operators by magic method

These features are interesting, but most programmers think it makes it more difficult to read other people’s code.

Go forces you to use the most basic things, which makes it easy to read someone else’s code.

Note: Of course, “easy” depends on the specific project. If you want to create a basic CRUD API, I still recommend using Django and DRF or Rails

Reason 4 — Concurrent and Channels

As a programming language, Go has always kept it as simple as possible. It doesn’t introduce too many new concepts because the goal is to create an easy-to-use programming language. The only places it has innovation are Goroutines and Channels. Goroutines is Go’s lightweight threading solution, and Channels is the preferred way to interact with Goss.

Goroutines are very lightweight and require only a few kilobytes of additional memory. And because Goroutines are so lightweight, there can be hundreds or even thousands of Goroutines running at the same time.

You can use Channels to communicate between Goroutines. The Go runtime handles all the internal complexity. Goroutines and Channels-based concurrency schemes make it easy for applications to use all available CPU cores and process concurrent IoOs without complicating development. Running a function on Goroutinerequires requires very little fixed code compared to Python/Java. You just need to call the function using the keyword “go”:

package mainimport (
    "fmt"
    "time"
)func say(s string) {
    for i := 0; i < 5; i++ {
        time.Sleep(100 * time.Millisecond)
        fmt.Println(s)
    }}func main() {
    go say("world")
    say("hello")
}

https://tour.golang.org/concurrency/1

Go’s concurrency solution is easy to use. This is a very interesting scenario compared to Node, where developers have to pay close attention to how asynchronous code is handled.

Another focus of Go concurrency is competition detection. It makes it easy for applications to know if there are any competing conditions in asynchronous code.

Here are a few great resources to learn about Go and Channels:

Reason 5 — Compilation speed is fast

The largest micro-service project written with Go can be compiled in just 6 seconds. Go’s fast compilation speed is its main productivity compared to the turtle-speed compilation in languages such as Java and C.

Reason 6 — The capabilities of the component team

Let’s start with this data: Go developers don’t have as many as C and Java. According to StackOverflow, 38% of developers use Java,19.3% use C, but only 4.6% use Go. GitHub data shows a similar trend: Go is more widely used than languages such as Erlang, Scala, and Elixir, but not as popular as Java and C.

Fortunately, Go is a very simple and easy-to-learn language. It only provides the basic functionality you need, and no other extras. It introduces new concepts such as”defer”statements and built-in “go routines” and Channels concurrent management. Any Python, Elixir, C, Scala, or Java developer on the team can learn to program with Go in a month because Go is so simple.

Compared to other languages, we found it easier to build a Go development team. This is a very important advantage if you recruit in a highly competitive environment, such as Boulder and Amsterdam.

Reason 7 — A strong ecosystem

The ecosystem is important for a team of our size (about 20 people). If you have to reinvent all the features, you simply can’t create value for your customers. Go is a great support for the tools we use frequently. For example, Redis, RabbitMQ, PostgreSQL, template resolution, task scheduling, expression resolution, and DBRocks can all use out-of-the-box libraries.

Go has a huge ecosystem advantage over other newer languages, such as Rust or Elixir. Although it can’t be compared to Java, Python, or Node, you’ll find high-quality packages available for many basic requirements.

Cause 8 — Gofmt, forced code formatting

Gofmt is an excellent command-line program that is built into the Go compiler and is used to format code. In terms of functionality, it is similar to Python’s autopep 8. Most of us don’t like to argue about tabs and spaces, but the goal of formatting is consistent and the actual format standard sittines don’t matter. Gofmt formats the code in a formal way to avoid all these arguments.

Cause 9 — gRPC and Protocol Buffers

Go provides first-class support for Protocol Buffers and gRPC. These two tools combine perfectly to build a microserver that communicates through RPC. You only need to edit an inventory file that defines the RPC call and its parameters, from which the service side and client can automatically generate the appropriate code. This is not only fast, but also less network footprint, is more convenient to use.

Client code in other languages, such as C , Java, Python, and Ruby, can be generated based on the same manifest file. In this way, there is no conflict with the internal REST interface, and we don’t have to write almost the same client-side and service-side code every time.

Disadvantages of using Golang

Cons1 — Lack of framework

Go doesn’t have a staple framework, such as Ruby’s Rails, Python or Django or PHP’s Laravel. The topic has been hotly debated in the Go community, where many argue that a ready-made framework should not be used to start a project. In some cases, I fully agree with that. However, if you want to build a simple CRUD API, it will be easier to use Django/DJRF, Rails Laravel, or Phoenix.

Cons2 — Error handling

Go handles errors by simply returning errors from a function. Although this scenario is available, it is easy to lose the scope of the error, making it difficult to provide valuable error messages to the user. The error pack solves this problem byreturningthe context of the error and the error stack.

There is also a problem, it is easy to forget to deal with errors. While static analysis tools such as errcheck and megacheck can avoid these errors, it always feels not perfect. Maybe we should expect a language-level error handling scheme.

Cons 3 — Package management

Go’s package management is not perfect. By default, it has no way to specify a specific version of the dependency, and it cannot create a repeatable build scenario. Python, Node, and Ruby all have better package management systems. However, Go’s package management can be made easier if the right tools are used.

You can use Dep to manage dependencies, which specify and pin versions. In addition, we offer an open source tool called VirtualGo for multi-project management.

Python vs Go

We did an interesting experiment and rewritten the original Python-written feed with Go. Take a look at an example of this sorting method:

{
    "functions": {
        "simple_gauss": {
            "base": "decay_gauss",
            "scale": "5d",
            "offset": "1d",
            "decay": "0.3"
        },
        "popularity_gauss": {
            "base": "decay_gauss",
            "scale": "100",
            "offset": "5",
            "decay": "0.5"
        }
    },
    "defaults": {
        "popularity": 1
    },
    "score": "simple_gauss(time)*popularity"
}

Both Python and Go codes need to do the following to support this sorting method:

Parsing the score expression, converting “simple_gauss” to a function, entering the activity, and outputting the score.
Create functions from the JSON configuration. For example, we want simple_gauss to call “decay_gauss” at a scale of 5 days, offset by 1 day, and attenuate factor 0.3.
Parsing the “defaults” configuration takes the default value when a field does not have a value.
Start using the function from step 1 to rate all the activities in the feed.

It took about three days to develop the Sort code for the Python version, including writing code, unit testing, and documentation. Next, it took us about 2 weeks to optimize the code. One of the optimizations is to convert the score expression (simple_gauss(time) to an abstract syntax tree. We also implemented caching logic that can be used to predict fractions.

By contrast, it took about four days to develop the Go version of this code, and there was no need to further optimize performance at a later stage. As a result, although Python developed faster in the first places, the Go version ultimately required less effort. Another advantage is that the Go code is about 40 times faster than our highly optimized Python code.

Of course, this is just a simple example of performance improvement we’ve encountered since we switched to Go:

The sort code was the first item I wrote with Go.
The Go code was written after the Python code, so the project is understood in greater depth
Go’s Expression Resolution Library Is Higher Quality

Your experience may be different. Some of the other components of our system need more time to build with Go than Python. As a general trend, we see that writing Go code takes more effort. However, it takes less time to optimize the performance of your code.

Elixir vs Go

Another language we want to evaluate is Elixir. Elixir is a fascinating language built on the Erlang virtual machine. I talked about it because one of our project team is well versed in the language.

For this reason, we noticed that Go’s original performance was better. Both Go and Elixir support thousands of concurrent requests. However, if you look at the performance of individual requests, Go is much faster. Another reason we chose Go is the ecosystem. Go has a more mature library for the components we need, and Elixir is not yet suitable for production. At the same time, it is difficult to train and find developers using Elixir to work with Elixir.

Conclusion

Go is a very high-performance language, and support for concurrency is very strong. It’s almost as fast as C and Java. Although Go compiles more slowly than Python or Ruby, you can save a lot of time optimizing your code.

Go has a huge ecosystem for novices and is easy to learn to use, it has ultra-fast performance and strong support in terms of concurrency, and a very efficient development environment. These features make Go the most suitable choice for developers.

If you want to learn more about Go, please read the article listed below. To learn more about Stream, browse this interactive tutorial.