Click here to Skip to main content
14,035,479 members
Click here to Skip to main content
Add your own
alternative version

Tagged as

Stats

950 views
Posted 14 Apr 2019
Licenced CPOL

Go: Checking Public Repositories List in Github. Go Slices Comparison. The First Golang Experience.

, 14 Apr 2019
Rate this:
Please Sign up or sign in to vote.
The task is to write a tool which will be started from a Jenkin's job by a cron to check an organization's public repositories list in the Github.

The task is to write a tool which will be started from a Jenkin’s job by a cron and will check an organization’s public repositories list in the Github.

Then it has to compare received repositories list with a predefined list with allowed repositories and if they will be not equal – send an alert to a Slack channel.

The idea is to have such a check in case developers will accidentally create a public repository instead of private or will change a private repository to the public and get a notification about such issue.

It could be written with bash and curl, or Python with urllib and use Github API directly, but I want to have some Go experience so will use it.

In fact – this is my very first self-written Golang code which I didn’t use a lot and even don’t know its syntax, but will try to use some existing knowledge with C/Python, add some logic and of course with the Google’s help and write something workable.

To work with API Github, the go-github package will be used.

Let’s begin.

Add imports, the function was created automatically with the vim-go plugin:

package main
    
import (
    "fmt"
    "github.com/google/go-github/github"
)

func main() {
    fmt.Println("vim-go")
}

Add $GOPATH:

$ sudo mkdir /usr/local/go
$ sudo chown setevoy:setevoy /usr/local/go/
$ export GOPATH=/usr/local/go && export GOBIN=/usr/local/go/bin

Install package:

$ go get
_/home/setevoy/Scripts/Go/GTHB
./gitrepos.go:5:2: imported and not used: "github.com/google/go-github/github"

Getting Repositories List From Github

Copy and paste the first example from the package’s README, an organization name will be taken from the $GITHUB_ORG_NAMEand let’s try to get something.

Set the variable:

$ export GITHUB_ORG_NAME="rtfmorg"

And code:

package main

import (
    "context"
    "fmt"
    "github.com/google/go-github/github"
)
 
func main() {

    client := github.NewClient(nil)

    opt := &github.RepositoryListByOrgOptions{Type: "public"}
    repos, _, _ := client.Repositories.ListByOrg(context.Background(), 
                   os.Getenv("GITHUB_ORG_NAME"), opt)

    fmt.Printf(repos)
}

In the original example, organization name was hardcoded but here we will take it from a variable which will be set in a Jenkin’s job using os.Getenv.

Add context and os imports.

Run the code:

$ go run go-github-public-repos-checker.go
command-line-arguments
./go-github-public-repos-checker.go:17:15: 
  cannot use repos (type []*github.Repository) as type string in argument to fmt.Printf

Errr…

Okay.

I thought client.Repositories.ListByOrg will return actually list – just because of the List in its name. 🙂

Check what data type we have in the repos object. Use reflect:

package main

import (
    "os"
    "context"
    "fmt"
    "reflect"
    "github.com/google/go-github/github"
)

func main() {

    client := github.NewClient(nil)

    opt := &github.RepositoryListByOrgOptions{Type: "public"}
    repos, _, _ := client.Repositories.ListByOrg(context.Background(), 
                   os.Getenv("GITHUB_ORG_NAME"), opt)

    fmt.Println(reflect.TypeOf(repos).String())
}

Run:

$ go run go-github-public-repos-checker.go
[]*github.Repository

Good… It’s really list ([]), apparently pointed to a structure – github.Repository.

Let’s check with the go doc:

go doc github.Repository
type Repository struct {
ID               *int64           `json:"id,omitempty"`
NodeID           *string          `json:"node_id,omitempty"`
Owner            *User            `json:"owner,omitempty"`
Name             *string          `json:"name,omitempty"`
FullName         *string          `json:"full_name,omitempty"`
Description      *string          `json:"description,omitempty"`
...

Yup – it’s structure and also we see all its fields.

To disable the “imported and not used: “reflect” message – set it as _reflect:

package main

import (
    "os"
    "context"
    "fmt"
    _"reflect"
    "github.com/google/go-github/github"
)

func main() {

    client := github.NewClient(nil)

    opt := &github.RepositoryListByOrgOptions{Type: "public"}
    repos, _, _ := client.Repositories.ListByOrg(context.Background(), 
                   os.Getenv("GITHUB_ORG_NAME"), opt)

    for _, repo := range repos {
        fmt.Printf("Repo: %s\n", *repo.Name)
    }
}

Here from the repos list, we are getting an element’s ID and its value.

Can add IDs as well:

...
    for id, repo := range repos {
        fmt.Printf("%d Repo: %s\n", id, *repo.Name)
    }
...
$ go run go-github-public-repos-checker.go
0 Repo: org-repo-1-pub
1 Repo: org-repo-2-pub

Comparing Lists in Go

Now need to add another one list which will keep repositories list from Github.

Allowed to be public repositories will be passed from Jenkins in a list view separated by spaces:

$ export ALLOWED_REPOS="1 2"

Attempt Number One

Create allowedRepos of the string type and save allowed public repositories list in it:

...
    allowedRepos := []string{os.Getenv("ALLOWED_REPOS")}

    for id, repo := range repos {
        fmt.Printf("%d Repo: %s\n", id, *repo.Name)
    }

    fmt.Printf("Allowed repos: %s\n", allowedRepos) 
...

But here is an issue which I faced a bit later when I started comparing lists. We will see it shortly.

Run the code:

$ go run go-github-public-repos-checker.go
0 Repo: org-repo-1-pub
1 Repo: org-repo-2-pub
Allowed repos: [1 2]

Okay – it works.

Now, we need to make checks between two repositories lists – repos with repositories from Github and allowedRepos.

The first solution was next:

...
    allowedRepos := []string{os.Getenv("ALLOWED_REPOS")}
    fmt.Printf("Allowed repos: %s\n", allowedRepos)

    for id, repo := range repos {
        for _, i := range allowedRepos {
            if i == *repo.Name {
                fmt.Printf("Index: %d, repo %s found in Allowed as %s\n", id, *repo.Name, i)
            } else {
                fmt.Printf("ALARM: repo %s was NOT found in Allowed!\n", *repo.Name)
            }
        }
    }
...

Run:

$ go run go-github-public-repos-checker.go
Allowed repos: [1 2]
ALARM: repo org-repo-1-pub was NOT found in Allowed!
ALARM: repo org-repo-2-pub was NOT found in Allowed!

Looks like it works?

Repositories with the 1 and 2 names != org-repo-1-pub and org-repo-2-pub.

But the problem which might be already obvious for some readers appeared when I did a “back-check”, i.e., when I set $ALLOWED_REPOS with real names to get the OK result:

$ export ALLOWED_REPOS="org-repo-1-pub org-repo-2-pub"

Check:

$ go run go-github-public-repos-checker.go
Allowed repos: [org-repo-1-pub org-repo-2-pub]
ALARM: repo org-repo-1-pub was NOT found in Allowed!
ALARM: repo org-repo-2-pub was NOT found in Allowed!

And why?

Because in the allowedRepos, we have not a list aka slice and Go – but just a string.

Let’s add the i variable’s output and indexes numbers:

...
    for r_id, repo := range repos {
        for a_id, i := range allowedRepos {
            fmt.Printf("ID: %d Type: %T Value: %s\n", a_id, allowedRepos, i)
            if i == *repo.Name {
                fmt.Printf("Index: %d, repo %s found in Allowed as %s\n", r_id, *repo.Name, i)
            } else {
                fmt.Printf("ALARM: repo %s was NOT found in Allowed!\n", *repo.Name)
            }
        }
    }
...

Check:

$ go run go-github-public-repos-checker.go
Allowed repos: [org-repo-1-pub org-repo-2-pub]
ID: 0 Type: []string Value: org-repo-1-pub org-repo-2-pub
ALARM: repo org-repo-1-pub was NOT found in Allowed!
ID: 0 Type: []string Value: org-repo-1-pub org-repo-2-pub
ALARM: repo org-repo-2-pub was NOT found in Allowed!

allowedRepos is really a list but with the only 0 element which keeps the “org-repo-1-pub org-repo-2-pub” value.

Attempt Number Two

To make this work – we need to convert the allowedRepos to a real list.

Let’s use the strings package:

...
    // allowedRepos := []string{os.Getenv("ALLOWED_REPOS")}
    allowedRepos := strings.Fields(os.Getenv("ALLOWED_REPOS"))
    fmt.Printf("Allowed repos: %s\n", allowedRepos)
    
    for r_id, repo := range repos {
        for a_id, i := range allowedRepos {
            fmt.Printf("ID: %d Type: %T Value: %s\n", a_id, allowedRepos, i)
            if i == *repo.Name {
                fmt.Printf("Index: %d, repo %s found in Allowed as %s\n", r_id, *repo.Name, i)
            } else {
                fmt.Printf("ALARM: repo %s was NOT found in Allowed!\n", *repo.Name)
            }
        }
    }
...

allowedRepos now will be filled with the strings.Fields().

Check result:

$ go run go-github-public-repos-checker.go
Allowed repos: [org-repo-1-pub org-repo-2-pub]
ID: 0 Type: []string Value: org-repo-1-pub
Index: 0, repo org-repo-1-pub found in Allowed as org-repo-1-pub
ID: 1 Type: []string Value: org-repo-2-pub
ALARM: repo org-repo-1-pub was NOT found in Allowed!
ID: 0 Type: []string Value: org-repo-1-pub
ALARM: repo org-repo-2-pub was NOT found in Allowed!
ID: 1 Type: []string Value: org-repo-2-pub
Index: 1, repo org-repo-2-pub found in Allowed as org-repo-2-pub

Okay – much better now but this also will not work as there are some “false-positive” results because each repos‘s element is compared with each allowedRepos‘s element.

Attempt Number Three

Let’s rewrite it and now let’s try to use a repo‘s indexes to chose an element from the allowedRepos:

...
    for r_id, repo := range repos {
        fmt.Printf("%d %s\n", r_id, *repo.Name)
        if *repo.Name != allowedRepos[r_id] {
            fmt.Printf("ALARM: repo %s was NOT found in Allowed!\n", *repo.Name)
        } else {
            fmt.Printf("Repo %s found in Allowed\n", *repo.Name)
        }
    }
...

I.e., from the repos, we getting an element’s ID and then checking the allowedRepos‘s element with the same ID.

Run:

$ go run go-github-public-repos-checker.go
Allowed repos: [org-repo-1-pub org-repo-2-pub]
0 org-repo-1-pub
Repo org-repo-1-pub found in Allowed
1 org-repo-2-pub
Repo org-repo-2-pub found in Allowed

Nice!

But another issue can happen now…

What if the order in both lists will differ?

Set instead of the:

$ export ALLOWED_REPOS="org-repo-1-pub org-repo-2-pub"

Repositories names in the reversed order:

$ export ALLOWED_REPOS="org-repo-2-pub org-repo-1-pub"

Check:

$ go run go-github-public-repos-checker.go
Allowed repos: [org-repo-2-pub org-repo-1-pub]
0 org-repo-1-pub
ALARM: repo org-repo-1-pub was NOT found in Allowed!
1 org-repo-2-pub
ALARM: repo org-repo-2-pub was NOT found in Allowed!

D’oh!

Attempt Number Four

The next idea was to use the reflect module and its DeepEqual() function.

Add one more list called actualRepos, add repositories taken from Github with the append and then compare them:

...
    allowedRepos := strings.Fields(os.Getenv("ALLOWED_REPOS"))
    var actualRepos []string

    for _,  repo := range repos {
        actualRepos = append(actualRepos, *repo.Name)
    }

    fmt.Printf("Allowed: %s\n", allowedRepos)
    fmt.Printf("Actual: %s\n", actualRepos)

    fmt.Println("Slice equal: ", reflect.DeepEqual(allowedRepos, actualRepos))
...

Run:

$ go run go-github-public-repos-checker.go
Allowed: [org-repo-2-pub org-repo-1-pub]
Actual: [org-repo-1-pub org-repo-2-pub]
Slice equal:  false

And again no… Although if revert order back – it will work:

$ export ALLOWED_REPOS="org-repo-1-pub org-repo-2-pub"
go run go-github-public-repos-checker.go
Allowed: [org-repo-1-pub org-repo-2-pub]
Actual: [org-repo-1-pub org-repo-2-pub]
Slice equal:  true

So, we need to go back to the loop’s variant but to somehow check repo‘s value in the whole allowedRepos list and only if it’s absent – create an alert.

Attempt Number Five

The solution was next: create a dedicated function which will check all allowedRepos‘s elements in a loop and will return true if the value will be found and false otherwise. Then – we can use this in the main()‘s loop.

Let’s try.

Create a isAllowedRepo()function which will accept two arguments – a repository name to be checked and the allowed repositories list and will return a boolean value:

...
func isAllowedRepo(repoName string, allowedRepos []string) bool {

    for _, i := range allowedRepos {
        if i == repoName {
            return true
        }
    }

    return false
}
...

Then in the main() – run a loop over all repos‘s elements, pass them one by one to the isAllowedRepo() and then print a result:

...
    for _, repo := range repos {
        fmt.Printf("Checking %s\n", *repo.Name)
        fmt.Println(isAllowedRepo(*repo.Name, allowedRepos))
    }
...

Let’s test.

First, restore allowed repositories list in the initial order:

$ export ALLOWED_REPOS="org-repo-1-pub org-repo-2-pub"
go run go-github-public-repos-checker.go
Checking org-repo-1-pub
true
Checking org-repo-2-pub
true

Good!

And reversed order:

$ export ALLOWED_REPOS="org-repo-2-pub org-repo-1-pub"
go run go-github-public-repos-checker.go
Checking org-repo-1-pub
true
Checking org-repo-2-pub
true

Still works…

Now remove one of the allowed repositories:

$ export ALLOWED_REPOS="org-repo-2-pub"
go run go-github-public-repos-checker.go
Checking org-repo-1-pub
false
Checking org-repo-2-pub
true

Great!

Now when we will get false – we can raise an alert.

The whole code now. Leaving “as is” just for history:

package main
import (
    "os"
    "context"
    "fmt"
    _"reflect"
    "strings"
    "github.com/google/go-github/github"
)

func isAllowedRepo(repoName string, allowedRepos []string) bool {

    for _, i := range allowedRepos {
        if i == repoName {
            return true
        }
    }

    return false
}

func main() {

    client := github.NewClient(nil)

    opt := &github.RepositoryListByOrgOptions{Type: "public"}
    repos, _, _ := client.Repositories.ListByOrg(context.Background(), 
                   os.Getenv("GITHUB_ORG_NAME"), opt)

    // allowedRepos := []string{os.Getenv("ALLOWED_REPOS")}
    allowedRepos := strings.Fields(os.Getenv("ALLOWED_REPOS"))
//  var actualRepos []string

/*
    for _,  repo := range repos {
        actualRepos = append(actualRepos, *repo.Name)
    }

    fmt.Printf("Allowed: %s\n", allowedRepos)
    fmt.Printf("Actual: %s\n", actualRepos)

    fmt.Println("Slice equal: ", reflect.DeepEqual(allowedRepos, actualRepos))
*/

    for _, repo := range repos {
        fmt.Printf("Checking %s\n", *repo.Name)
        fmt.Println(isAllowedRepo(*repo.Name, allowedRepos))
    }

/*
    for r_id, repo := range repos {
        for _, i := range allowedRepos {
            fmt.Printf("Checking %s and %s\n", *repo.Name, i)
            if i == *repo.Name {
                fmt.Printf("Index: %d, repo %s found in Allowed as %s\n", r_id, *repo.Name, i)
                break
            } else {
                fmt.Printf("ALARM: repo %s was NOT found in Allowed!\n", *repo.Name)
            }
        }
    }
*/

/*
    for r_id, repo := range repos {
        fmt.Printf("%d %s\n", r_id, *repo.Name)
        if *repo.Name != allowedRepos[r_id] {
            fmt.Printf("ALARM: repo %s was NOT found in Allowed!\n", *repo.Name)
        } else {
            fmt.Printf("Repo %s found in Allowed\n", *repo.Name)
        }
    }
*/

/*
    for r_id, repo := range repos {
        for a_id, i := range allowedRepos {
            fmt.Printf("ID: %d Type: %T Value: %s\n", a_id, allowedRepos, i)
            if i == *repo.Name {
                fmt.Printf("Index: %d, repo %s found in Allowed as %s\n", r_id, *repo.Name, i)
            } else {
                fmt.Printf("ALARM: repo %s was NOT found in Allowed!\n", *repo.Name)
            }
        }
    }
*/
}

Golang Slack

And the final thing is to add a Slack notification.

Use ashwanthkumar/slack-go-webhook here.

Configure Slack’s WebHook, get its URL.

Add a new sendSlackAlarm() function which will accept two arguments – repository name and its URL:

...
func sendSlackAlarm(repoName string, repoUrl string) {
    webhookUrl := os.Getenv("SLACK_URL")

    text := fmt.Sprintf(":scream: ALARM: repository *%s* was NOT found in Allowed!", repoName)

    attachment := slack.Attachment{}
    attachment.AddAction(slack.Action{Type: "button", Text: "RepoURL", 
                                      Url: repoUrl, Style: "danger"}) 

    payload := slack.Payload{
        Username:    "Github checker",
        Text:        text,
        Channel:     os.Getenv("SLACK_CHANNEL"),
        IconEmoji:   ":scream:",
        Attachments: []slack.Attachment{attachment},
    }

    err := slack.Send(webhookUrl, "", payload)
    if len(err) > 0 {
        fmt.Printf("error: %s\n", err)
    }
}
...

Add the sendSlackAlarm() execution to the main() if the isAllowedRepo() returned false:

...
    for _, repo := range repos {
        fmt.Printf("\nChecking %s\n", *repo.Name)
        if isAllowedRepo(*repo.Name, allowedRepos) {
            fmt.Printf("OK: repo %s found in Allowed\n", *repo.Name)
        } else {
            fmt.Printf("ALARM: repo %s was NOT found in Allowed!\n", *repo.Name)
            sendSlackAlarm(*repo.Name, *repo.HTMLURL)
        }
    }
...

repo.HTMLURL we found from the go doc github.Repository.

Add $SLACK_URL and $SLACK_CHANNEL environment variables:

$ export SLACK_URL="https://hooks.slack.com/services/T1641GRB9/BA***WRE"
$ export SLACK_CHANNEL="#general"

Restore the full repositories list on reversed order:

$ export ALLOWED_REPOS="org-repo-2-pub org-repo-1-pub"

Check:

$ go run go-github-public-repos-checker.go
Checking org-repo-1-pub
OK: repo org-repo-1-pub found in Allowed
Checking org-repo-2-pub
OK: repo org-repo-2-pub found in Allowed

Okay…

Remove one allowed:

$ export ALLOWED_REPOS="org-repo-2-pub"

Check again:

$ go run go-github-public-repos-checker.go
Checking org-repo-1-pub
ALARM: repo org-repo-1-pub was NOT found in Allowed!
Checking org-repo-2-pub
OK: repo org-repo-2-pub found in Allowed

And Slack notification:

Done!

The script is available in the setevoy-tools Github repository.

Similar Posts

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

setevoy4
Engineer
Ukraine Ukraine
DevOps, cloud and infrastructure engineer. Love Linux, OpenSource, and AWS.

You may also be interested in...

Pro

Comments and Discussions

 
-- There are no messages in this forum --
Permalink | Advertise | Privacy | Cookies | Terms of Use | Mobile
Web01 | 2.8.190424.1 | Last Updated 15 Apr 2019
Article Copyright 2019 by setevoy4
Everything else Copyright © CodeProject, 1999-2019
Layout: fixed | fluid