Concurrent File Downloader in Go: Utilizing Goroutines for Efficiency

Concurrent File Downloader in Go: Utilizing Goroutines for Efficiency

Introduction

One of Go’s standout features is its ability to perform tasks concurrently with minimal effort. In this post, we’ll build a concurrent file downloader that fetches multiple files in parallel using Goroutines and a WaitGroup to synchronize completion.

The Concurrent Downloader

We use sync.WaitGroup to wait for all our download Goroutines to finish their work before closing the application.

package main

import (
    "fmt"
    "io"
    "net/http"
    "os"
    "sync"
)

func downloadFile(url string, wg *sync.WaitGroup) {
    defer wg.Done() // Signal completion

    resp, err := http.Get(url)
    if err != nil {
        fmt.Printf("Error downloading %s: %s\n", url, err)
        return
    }
    defer resp.Body.Close()

    // Create a local file
    out, err := os.Create("downloaded_" + url[len(url)-5:])
    if err != nil {
        fmt.Println("Error creating file:", err)
        return
    }
    defer out.Close()

    io.Copy(out, resp.Body)
    fmt.Printf("Downloaded: %s\n", url)
}

func main() {
    urls := []string{
        "https://example.com/file1.txt",
        "https://example.com/file2.txt",
    }

    var wg sync.WaitGroup

    for _, url := range urls {
        wg.Add(1)
        go downloadFile(url, &wg)
    }

    wg.Wait() // Wait for all downloads to finish
    fmt.Println("All downloads completed.")
}

Why It’s Efficient

By wrapping each downloadFile function call in a go keyword, we launch a new, lightweight thread (Goroutine) for each file. This means if one download is slow, it won’t block the others, significantly reducing the total time required compared to downloading them sequentially.

Conclusion

Concurrency in Go is a powerful tool for performance optimization. Using Goroutines to handle I/O-bound tasks like file downloads is an idiomatic way to speed up your applications.

Suggested Reading

• Go Documentation: sync package
• Effective Go: Concurrency