Browse Source

Convert to gpm-style service (#11)

This is a **massively** breaking change from `v0.2.0`.
For consumers there will be no difference, `go-get` and `git-import` are both still supported.
The change will be for the admin regarding how package management works.

Prior to merging, `v0.2.0` should be moved to another branch in case myself or another party wants to continue with that style of service.

This version follows a similar implementation to [gpm](https://gitea.com/jolheiser/gpm) (and indeed, some code was copied nearly line-by-line)

-----

Vanity runs as a service, same as before. However, rather than automatic cron-style updates using a third-party API, now the service owner uses their local `vanity` binary with a matching `token` to...
* `vanity add` a new package
* `vanity update` an existing package
* `vanity remove` a package

This allows much finer control over which packages are in the service and should required almost no downtime once the service is started other than to update the service itself.

As well, it allows mixing of git providers.

<small>There's also an SDK, which is nice to have.</small>

Co-authored-by: jolheiser <john.olheiser@gmail.com>
Reviewed-on: https://gitea.com/jolheiser/vanity/pulls/11
Co-authored-by: John Olheiser <john.olheiser@gmail.com>
Co-committed-by: John Olheiser <john.olheiser@gmail.com>
pull/12/head v0.4.0
John Olheiser 6 months ago
parent
commit
cf984234fd
  1. 2
      .gitignore
  2. 24
      Makefile
  3. 72
      README.md
  4. 25
      api/package.go
  5. 3
      api/version.go
  6. 69
      cmd/add.go
  7. 181
      cmd/cmd.go
  8. 14
      cmd/flags/flags.go
  9. 67
      cmd/remove.go
  10. 55
      cmd/server.go
  11. 76
      cmd/update.go
  12. 28
      contrib/contrib.go
  13. 19
      contrib/vanity.service
  14. 84
      database/database.go
  15. 16
      database/errors.go
  16. 3
      docker/docker-compose.yml
  17. 105
      flags/config.go
  18. 236
      flags/flags.go
  19. 53
      go-vanity/client.go
  20. 3
      go-vanity/go.mod
  21. 104
      go-vanity/package.go
  22. 45
      go-vanity/source.go
  23. 194
      go-vanity/vanity_test.go
  24. 16
      go.mod
  25. 399
      go.sum
  26. 27
      main.go
  27. 71
      router/api.go
  28. 110
      router/cache.go
  29. 30
      router/cron.go
  30. 228
      router/router.go
  31. 175
      router/router_test.go
  32. 22
      router/templates.go
  33. 32
      router/templates/base.tmpl
  34. 25
      router/templates/foot.tmpl
  35. 21
      router/templates/head.tmpl
  36. 26
      router/templates/import.tmpl
  37. 36
      router/templates/index.tmpl
  38. 5
      router/templates/vanity.tmpl
  39. 95
      service/gitea.go
  40. 90
      service/github.go
  41. 85
      service/gitlab.go
  42. 23
      service/off.go
  43. 75
      service/service.go
  44. 21
      vanity.service

2
.gitignore

@ -4,4 +4,4 @@
# Vanity
/vanity
/vanity.exe
.vanity.toml
/vanity.db

24
Makefile

@ -4,15 +4,29 @@ VERSION ?= $(shell git describe --tags --always | sed 's/-/+/' | sed 's/^v//')
.PHONY: build
build:
$(GO) build -ldflags '-s -w -X "go.jolheiser.com/vanity/api.Version=$(VERSION)"'
$(GO) build -ldflags '-s -w -X "go.jolheiser.com/vanity/router.Version=$(VERSION)"'
.PHONY: fmt
fmt:
$(GO) fmt ./...
fmt: fmt-cli fmt-lib
.PHONY: test
test:
$(GO) test --race ./...
test: test-cli test-lib
.PHONY: fmt-cli
fmt-cli:
$(GO) fmt ./...
.PHONY: test-cli
test-cli:
$(GO) test -race ./...
.PHONY: fmt-lib
fmt-lib:
@cd go-vanity && $(GO) fmt ./...
.PHONY: test-lib
test-lib:
@cd go-vanity && $(GO) test -race ./...
.PHONY: vet
vet:

72
README.md

@ -4,90 +4,20 @@ A simple web service to serve [vanity Go imports](https://golang.org/cmd/go/#hdr
Vanity also supports [git-import](https://gitea.com/jolheiser/git-import).
## Configuration
When choosing a service, the default `base-url` will be the default server of that service:
| Service | Default |
|:-------:|:------------------:|
| Gitea | https://gitea.com |
| GitHub | https://github.com |
| GitLab | https://gitlab.com |
```
NAME:
vanity - Vanity Go Imports
USAGE:
vanity [global options] command [command options] [arguments...]
VERSION:
0.1.0+3-g6d7150e
COMMANDS:
help, h Shows a list of commands or help for one command
GLOBAL OPTIONS:
--config value Path to a config file [$VANITY_CONFIG]
--port value Port to run the vanity server on (default: 7777) [$VANITY_PORT]
--domain value Vanity domain, e.g. go.domain.tld [$VANITY_DOMAIN]
--service value Service type (Gitea, GitHub, GitLab) (default: "gitea") [$VANITY_SERVICE]
--base-url value Base URL to service [$VANITY_BASE_URL]
--namespace value Owner namespace [$VANITY_NAMESPACE]
--token value Access token [$VANITY_TOKEN]
--include value Repository names to include (regex) [$VANITY_INCLUDE]
--exclude value Repository names to exclude (regex) [$VANITY_EXCLUDE]
--private Include private repositories (default: false) [$VANITY_PRIVATE]
--fork Include forked repositories (default: false) [$VANITY_FORK]
--mirror Include mirrored repositories (default: false) [$VANITY_MIRROR]
--archive Include archived repositories (default: false) [$VANITY_ARCHIVE]
--override value Repository name to override (NAME=OVERRIDE) [$VANITY_OVERRIDE]
--interval value Interval between updating repositories (default: 15m0s) [$VANITY_INTERVAL]
--debug Debug logging (default: false) [$VANITY_DEBUG]
--help, -h show help (default: false)
--version, -v print the version (default: false)
```
## Docker
```sh
docker run \
--env VANITY_DOMAIN=go.domain.tld \
--env VANITY_NAMESPACE=<jolheiser> \
--env VANITY_TOKEN=<token> \
--publish 80:7777 \
--restart always
jolheiser/vanity:latest
```
## Overrides
Certain modules may not align perfectly with their repository name.
Overrides are available via config or by setting an environment variable `VANITY_OVERRIDE_PACKAGE=NAME`
## Config-only Mode
To run Vanity in config-only mode for packages, set `--service` to `off`.
## Manual Mode
To run Vanity without automatic updating, use `--manual`.
When running with manual-mode, the provided button or `/_/update` endpoint can be used once every `--interval`.
## Topic Lists
By setting `--topics`, anyone visiting the index page will see packages grouped by their topics.
Regardless of the setting, you can switch beteween list-view and topic-view with the provided button
or changing the URL between `?format=list` and `?format=topics`.
## API
In order to preserve namespaces for packages, Vanity's API uses the URL `/_/{endpoint}`
Vanity currently supports `/_/status` and `/_/update`, to get some status information and update the package cache respectively.
Check out the [SDK](go-vanity).
## License

25
api/package.go

@ -1,25 +0,0 @@
package api
import (
"fmt"
"strings"
)
type Package struct {
Name string `toml:"name"`
Description string `toml:"description"`
Branch string `toml:"branch"`
WebURL string `toml:"web_url"`
CloneHTTP string `toml:"clone_http"`
CloneSSH string `toml:"clone_ssh"`
Topics []string `toml:"topics"`
Private bool `toml:"-"`
Fork bool `toml:"-"`
Mirror bool `toml:"-"`
Archive bool `toml:"-"`
}
func (p *Package) Module(domain string) string {
return fmt.Sprintf("%s/%s", domain, strings.ToLower(p.Name))
}

3
api/version.go

@ -1,3 +0,0 @@
package api
var Version = "develop"

69
cmd/add.go

@ -0,0 +1,69 @@
package cmd
import (
"context"
"net/url"
"go.jolheiser.com/vanity/cmd/flags"
"go.jolheiser.com/vanity/database"
"go.jolheiser.com/vanity/go-vanity"
"github.com/AlecAivazis/survey/v2"
"github.com/urfave/cli/v2"
"go.jolheiser.com/beaver"
)
var Add = cli.Command{
Name: "add",
Aliases: []string{"a"},
Usage: "Add a package",
Flags: []cli.Flag{
&cli.BoolFlag{
Name: "force",
Aliases: []string{"f"},
Usage: "Overwrite existing package without prompt",
Destination: &flags.Force,
},
&cli.BoolFlag{
Name: "local",
Aliases: []string{"l"},
Usage: "local mode",
Destination: &flags.Local,
},
},
Before: localOrToken,
Action: doAdd,
}
func doAdd(_ *cli.Context) error {
pkg, err := pkgPrompt(vanity.Package{})
if err != nil {
return err
}
if flags.Local {
db, err := database.Load(flags.Database)
if err != nil {
return err
}
if err := db.PutPackage(pkg); err != nil {
return err
}
} else {
client := vanity.New(flags.Token, vanity.WithServer(flags.Server))
if err := client.Add(context.Background(), pkg); err != nil {
return err
}
}
beaver.Infof("Added %s", yellow.Format(pkg.Name))
return nil
}
func validURL(ans interface{}) error {
if err := survey.Required(ans); err != nil {
return err
}
_, err := url.Parse(ans.(string))
return err
}

181
cmd/cmd.go

@ -0,0 +1,181 @@
package cmd
import (
"context"
"errors"
"fmt"
"net/url"
"os"
"path/filepath"
"strings"
"go.jolheiser.com/vanity/cmd/flags"
"go.jolheiser.com/vanity/contrib"
"go.jolheiser.com/vanity/database"
"go.jolheiser.com/vanity/go-vanity"
"go.jolheiser.com/vanity/router"
"github.com/AlecAivazis/survey/v2"
"github.com/urfave/cli/v2"
"go.jolheiser.com/beaver/color"
)
var yellow = color.FgYellow
func New() *cli.App {
app := cli.NewApp()
app.Name = "vanity"
app.Usage = "Vanity Import URLs"
app.Version = router.Version
app.Commands = []*cli.Command{
&Add,
&Remove,
&Server,
&Update,
}
app.Flags = []cli.Flag{
&cli.StringFlag{
Name: "server",
Aliases: []string{"s"},
Usage: "vanity server to use",
Value: vanity.DefaultServer,
EnvVars: []string{"VANITY_SERVER"},
Destination: &flags.Server,
},
&cli.StringFlag{
Name: "token",
Aliases: []string{"t"},
Usage: "vanity auth token to use",
DefaultText: "${VANITY_TOKEN}",
EnvVars: []string{"VANITY_TOKEN"},
Destination: &flags.Token,
},
&cli.StringFlag{
Name: "database",
Aliases: []string{"d"},
Usage: "path to vanity database for server",
Value: dbPath(),
DefaultText: "`${HOME}/vanity.db` or `${BINPATH}/vanity.db`",
EnvVars: []string{"VANITY_DATABASE"},
Destination: &flags.Database,
},
&cli.BoolFlag{
Name: "systemd-service",
Usage: "Output example systemd service",
Destination: &flags.SystemdService,
Hidden: true,
},
}
app.Action = action
return app
}
func action(ctx *cli.Context) error {
if flags.SystemdService {
fmt.Println(contrib.SystemdService)
return nil
}
return cli.ShowAppHelp(ctx)
}
func dbPath() string {
fn := "vanity.db"
home, err := os.UserHomeDir()
if err != nil {
bin, err := os.Executable()
if err != nil {
return fn
}
return filepath.Join(filepath.Dir(bin), fn)
}
return filepath.Join(home, fn)
}
func localOrToken(_ *cli.Context) error {
if flags.Local && flags.Token == "" {
return errors.New("server interaction requires --token")
}
return nil
}
func listPackages() ([]vanity.Package, error) {
var pkgs []vanity.Package
if flags.Local {
db, err := database.Load(flags.Database)
if err != nil {
return pkgs, err
}
pkgs, err = db.Packages()
if err != nil {
return pkgs, err
}
} else {
client := vanity.New(flags.Token, vanity.WithServer(flags.Server))
info, err := client.Info(context.Background())
if err != nil {
return pkgs, err
}
pkgs = info.Packages
}
return pkgs, nil
}
func pkgPrompt(def vanity.Package) (vanity.Package, error) {
if def.Branch == "" {
def.Branch = "main"
}
var pkg vanity.Package
questions := []*survey.Question{
{
Name: "name",
Prompt: &survey.Input{Message: "Name", Default: def.Name},
Validate: survey.Required,
},
{
Name: "description",
Prompt: &survey.Multiline{Message: "Description", Default: def.Description},
Validate: survey.Required,
},
{
Name: "branch",
Prompt: &survey.Input{Message: "Branch", Default: def.Branch},
Validate: survey.Required,
},
{
Name: "weburl",
Prompt: &survey.Input{Message: "Web URL", Default: def.WebURL},
Validate: validURL,
},
}
if err := survey.Ask(questions, &pkg); err != nil {
return pkg, err
}
defHTTP, defSSH := def.CloneHTTP, def.CloneSSH
if def.WebURL != pkg.WebURL {
u, err := url.Parse(pkg.WebURL)
if err != nil {
return pkg, err
}
defHTTP = pkg.WebURL + ".git"
defSSH = fmt.Sprintf("git@%s:%s.git", u.Host, strings.TrimPrefix(u.Path, "/"))
}
questions = []*survey.Question{
{
Name: "clonehttp",
Prompt: &survey.Input{Message: "HTTP(S) CLone URL", Default: defHTTP},
Validate: validURL,
},
{
Name: "clonessh",
Prompt: &survey.Input{Message: "SSH CLone URL", Default: defSSH},
Validate: survey.Required,
},
}
if err := survey.Ask(questions, &pkg); err != nil {
return pkg, err
}
return pkg, nil
}

14
cmd/flags/flags.go

@ -0,0 +1,14 @@
package flags
var (
Server string
Domain string
Token string
Database string
Local bool
Force bool
Port int
SystemdService bool
)

67
cmd/remove.go

@ -0,0 +1,67 @@
package cmd
import (
"context"
"go.jolheiser.com/vanity/cmd/flags"
"go.jolheiser.com/vanity/database"
"go.jolheiser.com/vanity/go-vanity"
"github.com/AlecAivazis/survey/v2"
"github.com/urfave/cli/v2"
"go.jolheiser.com/beaver"
)
var Remove = cli.Command{
Name: "remove",
Aliases: []string{"rm"},
Usage: "Remove package(s)",
Before: localOrToken,
Action: doRemove,
}
func doRemove(_ *cli.Context) error {
pkgs, err := listPackages()
if err != nil {
return err
}
pkgSlice := make([]string, len(pkgs))
pkgMap := make(map[string]vanity.Package)
for idx, pkg := range pkgs {
pkgSlice[idx] = pkg.Name
pkgMap[pkg.Name] = pkg
}
pkgQuestion := &survey.Select{
Message: "Select package to remove",
Options: pkgSlice,
}
var pkgName string
if err := survey.AskOne(pkgQuestion, &pkgName); err != nil {
return err
}
pkg := vanity.Package{
Name: pkgName,
}
if flags.Local {
db, err := database.Load(flags.Database)
if err != nil {
return err
}
if err := db.RemovePackage(pkg.Name); err != nil {
return err
}
} else {
client := vanity.New(flags.Token, vanity.WithServer(flags.Server))
if err := client.Remove(context.Background(), pkg); err != nil {
return err
}
}
beaver.Infof("Removed %s", yellow.Format(pkgName))
return nil
}

55
cmd/server.go

@ -0,0 +1,55 @@
package cmd
import (
"errors"
"fmt"
"net/http"
"go.jolheiser.com/vanity/cmd/flags"
"go.jolheiser.com/vanity/database"
"go.jolheiser.com/vanity/router"
"github.com/urfave/cli/v2"
"go.jolheiser.com/beaver"
)
var Server = cli.Command{
Name: "server",
Aliases: []string{"web"},
Usage: "Start the vanity server",
Flags: []cli.Flag{
&cli.IntFlag{
Name: "port",
Aliases: []string{"p"},
Usage: "Port to run the vanity server on",
Value: 3333,
EnvVars: []string{"VANITY_PORT"},
Destination: &flags.Port,
},
&cli.StringFlag{
Name: "domain",
Aliases: []string{"d"},
Usage: "The Go module domain (e.g. go.jolheiser.com)",
EnvVars: []string{"VANITY_DOMAIN"},
Destination: &flags.Domain,
},
},
Action: doServer,
}
func doServer(_ *cli.Context) error {
if flags.Token == "" || flags.Domain == "" {
return errors.New("vanity server requires --token and --domain")
}
db, err := database.Load(flags.Database)
if err != nil {
beaver.Fatalf("could not load database at %s: %v", flags.Database, err)
}
beaver.Infof("Running vanity server at http://localhost:%d", flags.Port)
if err := http.ListenAndServe(fmt.Sprintf(":%d", flags.Port), router.New(flags.Token, db)); err != nil {
return err
}
return nil
}

76
cmd/update.go

@ -0,0 +1,76 @@
package cmd
import (
"context"
"go.jolheiser.com/vanity/cmd/flags"
"go.jolheiser.com/vanity/database"
"go.jolheiser.com/vanity/go-vanity"
"github.com/AlecAivazis/survey/v2"
"github.com/urfave/cli/v2"
"go.jolheiser.com/beaver"
)
var Update = cli.Command{
Name: "update",
Aliases: []string{"u"},
Usage: "Update a package",
Flags: []cli.Flag{
&cli.BoolFlag{
Name: "local",
Aliases: []string{"l"},
Usage: "local mode",
Destination: &flags.Local,
},
},
Before: localOrToken,
Action: doUpdate,
}
func doUpdate(_ *cli.Context) error {
pkgs, err := listPackages()
if err != nil {
return err
}
pkgSlice := make([]string, len(pkgs))
pkgMap := make(map[string]vanity.Package)
for idx, pkg := range pkgs {
pkgSlice[idx] = pkg.Name
pkgMap[pkg.Name] = pkg
}
pkgQuestion := &survey.Select{
Message: "Select package to update",
Options: pkgSlice,
}
var pkgName string
if err := survey.AskOne(pkgQuestion, &pkgName); err != nil {
return err
}
pkg, err := pkgPrompt(pkgMap[pkgName])
if err != nil {
return err
}
if flags.Local {
db, err := database.Load(flags.Database)
if err != nil {
return err
}
if err := db.PutPackage(pkg); err != nil {
return err
}
} else {
client := vanity.New(flags.Token, vanity.WithServer(flags.Server))
if err := client.Update(context.Background(), pkg); err != nil {
return err
}
}
beaver.Infof("Updated %s", yellow.Format(pkgName))
return nil
}

28
contrib/contrib.go

@ -0,0 +1,28 @@
package contrib
import (
_ "embed"
"os"
"strings"
)
//go:embed vanity.service
var SystemdService string
func init() {
bin, err := os.Executable()
if err != nil {
return
}
SystemdService = os.Expand(SystemdService, func(s string) string {
switch strings.ToUpper(s) {
case "BIN":
return bin
case "VANITY_TOKEN":
return os.Getenv("VANITY_TOKEN")
case "VANITY_DOMAIN":
return os.Getenv("VANITY_DOMAIN")
}
return ""
})
}

19
contrib/vanity.service

@ -0,0 +1,19 @@
[Unit]
Description=Vanity Go Imports
After=syslog.target
After=network.target
[Service]
RestartSec=2s
Type=simple
User=vanity
Group=vanity
ExecStart=${bin} server
Restart=always
# Required
Environment=VANITY_TOKEN=${VANITY_TOKEN}
Environment=VANITY_DOMAIN=${VANITY_DOMAIN}
[Install]
WantedBy=multi-user.target

84
database/database.go

@ -0,0 +1,84 @@
package database
import (
"bytes"
"encoding/json"
"os"
"path/filepath"
"go.jolheiser.com/vanity/go-vanity"
"go.etcd.io/bbolt"
)
var packageBucket = []byte("packages")
type Database struct {
db *bbolt.DB
}
func Load(dbPath string) (*Database, error) {
if err := os.MkdirAll(filepath.Dir(dbPath), os.ModePerm); err != nil {
return nil, err
}
db, err := bbolt.Open(dbPath, os.ModePerm, nil)
if err != nil {
return nil, err
}
return &Database{
db: db,
}, db.Update(func(tx *bbolt.Tx) error {
_, err := tx.CreateBucketIfNotExists(packageBucket)
return err
})
}
func (d *Database) Package(name string) (vanity.Package, error) {
var pkg vanity.Package
data, err := d.PackageJSON(name)
if err != nil {
return pkg, err
}
return pkg, json.NewDecoder(bytes.NewReader(data)).Decode(&pkg)
}
func (d *Database) PackageJSON(name string) (pkg []byte, err error) {
return pkg, d.db.View(func(tx *bbolt.Tx) error {
pkg = tx.Bucket(packageBucket).Get([]byte(name))
if pkg == nil {
return ErrPackageNotFound{
Name: name,
}
}
return nil
})
}
func (d *Database) Packages() (pkgs []vanity.Package, err error) {
return pkgs, d.db.View(func(tx *bbolt.Tx) error {
return tx.Bucket(packageBucket).ForEach(func(key, val []byte) error {
var pkg vanity.Package
if err := json.NewDecoder(bytes.NewReader(val)).Decode(&pkg); err != nil {
return err
}
pkgs = append(pkgs, pkg)
return nil
})
})
}
func (d *Database) PutPackage(pkg vanity.Package) error {
return d.db.Update(func(tx *bbolt.Tx) error {
data, err := json.Marshal(pkg)
if err != nil {
return err
}
return tx.Bucket(packageBucket).Put([]byte(pkg.Name), data)
})
}
func (d *Database) RemovePackage(name string) error {
return d.db.Update(func(tx *bbolt.Tx) error {
return tx.Bucket(packageBucket).Delete([]byte(name))
})
}

16
database/errors.go

@ -0,0 +1,16 @@
package database
import "fmt"
type ErrPackageNotFound struct {
Name string
}
func (e ErrPackageNotFound) Error() string {
return fmt.Sprintf("package not found: %s", e.Name)
}
func IsErrPackageNotFound(err error) bool {
_, ok := err.(ErrPackageNotFound)
return ok
}

3
docker/docker-compose.yml

@ -5,10 +5,7 @@ services:
image: jolheiser/vanity:latest
environment:
- VANITY_DOMAIN=go.domain.tld
- VANITY_NAMESPACE=<jolheiser>
- VANITY_TOKEN=<token>
#- VANITY_SERVICE=gitea
#- VANITY_BASE_URL=https://gitea.com
restart: always
ports:
- "80:7777"

105
flags/config.go

@ -1,105 +0,0 @@
package flags
import (
"os"
"strings"
"time"
"go.jolheiser.com/vanity/api"
"github.com/pelletier/go-toml"
"github.com/urfave/cli/v2"
"go.jolheiser.com/beaver"
)
type tomlConfig struct {
Port int `toml:"port"`
Domain string `toml:"domain"`
Service string `toml:"service"`
BaseURL string `toml:"base_url"`
Namespace string `toml:"namespace"`
Token string `toml:"token"`
Include []string `toml:"include"`
Exclude []string `toml:"exclude"`
Private bool `toml:"private"`
Fork bool `toml:"fork"`
Mirror bool `toml:"mirror"`
Archive bool `toml:"archive"`
Override []string `toml:"override"`
Interval time.Duration `toml:"interval"`
Debug bool `toml:"debug"`
Packages []*api.Package `toml:"packages"`
}
func setConfig(ctx *cli.Context) {
for _, env := range os.Environ() {
kv := strings.Split(env, "=")
if strings.HasPrefix(kv[0], "VANITY_OVERRIDES_") {
override := strings.ToLower(strings.TrimPrefix(kv[0], "VANITY_OVERRIDES_"))
Override[override] = kv[1]
}
}
var cfg tomlConfig
if configPath != "" {
beaver.Infof("Loading configuration from %s", configPath)
tree, err := toml.LoadFile(configPath)
if err != nil {
beaver.Errorf("Could not load configuration from %s: %v", configPath, err)
return
}
if err = tree.Unmarshal(&cfg); err != nil {
beaver.Errorf("Could not unmarshal configuration from %s: %v", configPath, err)
return
}
}
if !ctx.IsSet("port") && cfg.Port > 0 {
Port = cfg.Port
}
if !ctx.IsSet("domain") && cfg.Domain != "" {
Domain = cfg.Domain
}
if !ctx.IsSet("service") && cfg.Service != "" {
Service = cfg.Service
}
if !ctx.IsSet("base-url") && cfg.BaseURL != "" {
baseURL = cfg.BaseURL
}
if !ctx.IsSet("namespace") && cfg.Namespace != "" {
Namespace = cfg.Namespace
}
if !ctx.IsSet("token") && cfg.Token != "" {
Token = cfg.Token
}
if !ctx.IsSet("include") && len(cfg.Include) > 0 {
_ = include.Set(strings.Join(cfg.Include, ","))
}
if !ctx.IsSet("exclude") && len(cfg.Exclude) > 0 {
_ = exclude.Set(strings.Join(cfg.Exclude, ","))
}
if !ctx.IsSet("override") && len(cfg.Override) > 0 {
_ = override.Set(strings.Join(cfg.Override, ","))
}
if !ctx.IsSet("private") && cfg.Private {
Private = cfg.Private
}
if !ctx.IsSet("fork") && cfg.Fork {
Fork = cfg.Fork
}
if !ctx.IsSet("mirror") && cfg.Mirror {
Mirror = cfg.Mirror
}
if !ctx.IsSet("archive") && cfg.Archive {
Archive = cfg.Archive
}
if !ctx.IsSet("interval") && cfg.Interval.Seconds() > 0 {
Interval = cfg.Interval
}
if !ctx.IsSet("debug") && cfg.Debug {
Debug = cfg.Debug
}
ConfigPackages = cfg.Packages
}

236
flags/flags.go

@ -1,236 +0,0 @@
package flags
import (
"errors"
"fmt"
"net/url"
"regexp"
"strings"
"time"
"go.jolheiser.com/vanity/api"
"github.com/urfave/cli/v2"
"go.jolheiser.com/beaver"
)
var (
configPath string
baseURL string
include cli.StringSlice
exclude cli.StringSlice
override cli.StringSlice
Port int
Domain string
Service string
BaseURL *url.URL
Namespace string
Token string
Include []*regexp.Regexp
Exclude []*regexp.Regexp
Private bool
Fork bool
Mirror bool
Archive bool
Override = make(map[string]string)
Interval time.Duration
Manual bool
Topics bool
Debug bool
ConfigPackages []*api.Package
)
var Flags = []cli.Flag{
&cli.StringFlag{
Name: "config",
Usage: "Path to a config file",
EnvVars: []string{"VANITY_CONFIG"},
Destination: &configPath,
},
&cli.IntFlag{
Name: "port",
Usage: "Port to run the vanity server on",
Value: 7777,
EnvVars: []string{"VANITY_PORT"},
Destination: &Port,
},
&cli.StringFlag{
Name: "domain",
Usage: "Vanity domain, e.g. go.domain.tld",
EnvVars: []string{"VANITY_DOMAIN"},
Required: true,
Destination: &Domain,
},
&cli.StringFlag{
Name: "service",
Usage: "Service type (Gitea, GitHub, GitLab)",
Value: "gitea",
EnvVars: []string{"VANITY_SERVICE"},
Destination: &Service,
},
&cli.StringFlag{
Name: "base-url",
Usage: "Base URL to service",
EnvVars: []string{"VANITY_BASE_URL"},
Destination: &baseURL,
},
&cli.StringFlag{
Name: "namespace",
Usage: "Owner namespace",
EnvVars: []string{"VANITY_NAMESPACE"},
Destination: &Namespace,
},
&cli.StringFlag{
Name: "token",
Usage: "Access token",
EnvVars: []string{"VANITY_TOKEN"},
Destination: &Token,
},
&cli.StringSliceFlag{
Name: "include",
Usage: "Repository names to include (regex)",
EnvVars: []string{"VANITY_INCLUDE"},
Destination: &include,
},
&cli.StringSliceFlag{
Name: "exclude",
Usage: "Repository names to exclude (regex)",
EnvVars: []string{"VANITY_EXCLUDE"},
Destination: &exclude,
},
&cli.BoolFlag{
Name: "private",
Usage: "Include private repositories",
EnvVars: []string{"VANITY_PRIVATE"},
Destination: &Private,
},
&cli.BoolFlag{
Name: "fork",
Usage: "Include forked repositories",
EnvVars: []string{"VANITY_FORK"},
Destination: &Fork,
},
&cli.BoolFlag{
Name: "mirror",
Usage: "Include mirrored repositories",
EnvVars: []string{"VANITY_MIRROR"},
Destination: &Mirror,
},
&cli.BoolFlag{
Name: "archive",
Usage: "Include archived repositories",
EnvVars: []string{"VANITY_ARCHIVE"},
Destination: &Archive,
},
&cli.StringSliceFlag{
Name: "override",
Usage: "Repository name to override (NAME=OVERRIDE)",
EnvVars: []string{"VANITY_OVERRIDE"},
Destination: &override,
},
&cli.DurationFlag{
Name: "interval",
Usage: "Interval between updating repositories",
Value: time.Minute * 15,
EnvVars: []string{"VANITY_INTERVAL"},
Destination: &Interval,
},
&cli.BoolFlag{
Name: "manual",
Usage: "Disable cron and only update with endpoint",
EnvVars: []string{"VANITY_MANUAL"},
Destination: &Manual,
},
&cli.BoolFlag{
Name: "topics",
Usage: "Group projects by topic by default",
EnvVars: []string{"VANITY_TOPICS"},
Destination: &Topics,
},
&cli.BoolFlag{
Name: "debug",
Usage: "Debug logging",
EnvVars: []string{"VANITY_DEBUG"},
Destination: &Debug,
},
}
func Before(ctx *cli.Context) error {
setConfig(ctx)
var defaultURL string
var configOnly bool
switch strings.ToLower(Service) {
case "gitea":
defaultURL = "https://gitea.com"
case "github":
defaultURL = "https://github.com"
case "gitlab":
defaultURL = "https://gitlab.com"
case "off":
configOnly = true
beaver.Infof("Running in config-only mode")
defaultURL = "https://domain.tld"
default:
return errors.New("unrecognized service type")
}
if baseURL == "" {
baseURL = defaultURL
}
var err error
BaseURL, err = url.Parse(baseURL)
if err != nil {
return err
}
if !configOnly {
errs := make([]string, 0, 2)
if Namespace == "" {
errs = append(errs, "namespace")
}
if Token == "" {
errs = append(errs, "token")
}
if len(errs) > 0 {
return fmt.Errorf("%s is required with a service", strings.Join(errs, ", "))
}
}
Include = make([]*regexp.Regexp, len(include.Value()))
for idx, i := range include.Value() {
Include[idx] = regexp.MustCompile(i)
}
Exclude = make([]*regexp.Regexp, len(exclude.Value()))
for idx, e := range exclude.Value() {
Exclude[idx] = regexp.MustCompile(e)
}
if Manual {
beaver.Info("Running in manual mode")
}
if Debug {
beaver.Console.Level = beaver.DEBUG
}
beaver.Debugf("Port: %d", Port)
beaver.Debugf("Domain: %s", Domain)
beaver.Debugf("Service: %s", Service)
beaver.Debugf("Base URL: %s", baseURL)
beaver.Debugf("Namespace: %s", Namespace)
beaver.Debugf("Include: %s", include.Value())
beaver.Debugf("Exclude: %s", exclude.Value())
beaver.Debugf("Private: %t", Private)
beaver.Debugf("Fork: %t", Fork)
beaver.Debugf("Mirror: %t", Mirror)
beaver.Debugf("Archive: %t", Archive)
beaver.Debugf("Override: %s", override.Value())
beaver.Debugf("Interval: %s", Interval)
beaver.Debugf("Manual: %t", Manual)
return nil
}

53
go-vanity/client.go

@ -0,0 +1,53 @@
package vanity
import (
"context"
"io"
"net/http"
)
const (
DefaultServer = "https://go.jolheiser.com"
TokenHeader = "X-Vanity-Token"
)
type Client struct {
token string
server string
http *http.Client
}
func New(token string, opts ...ClientOption) *Client {
c := &Client{
token: token,
server: DefaultServer,
http: http.DefaultClient,
}
for _, opt := range opts {
opt(c)
}
return c
}
type ClientOption func(*Client)
func WithHTTP(client *http.Client) ClientOption {
return func(c *Client) {
c.http = client
}
}
func WithServer(server string) ClientOption {
return func(c *Client) {
c.server = server
}
}
func (c *Client) newRequest(ctx context.Context, method, url string, body io.Reader) (*http.Request, error) {
req, err := http.NewRequestWithContext(ctx, method, url, body)
if err != nil {
return nil, err
}
req.Header.Set(TokenHeader, c.token)
return req, nil
}

3
go-vanity/go.mod

@ -0,0 +1,3 @@
module go.jolheiser.com/vanity/go-vanity
go 1.16

104
go-vanity/package.go

@ -0,0 +1,104 @@
package vanity
import (
"bytes"
"context"
"encoding/json"
"fmt"
"net/http"
"strings"
)
type Info struct {
Version string `json:"version"`
NumPackages int `json:"num_packages"`
Packages []Package `json:"packages"`
}
type Package struct {
Name string `json:"name"`
Description string `json:"description"`
Branch string `json:"branch"`
WebURL string `json:"web_url"`
CloneHTTP string `json:"clone_http"`
CloneSSH string `json:"clone_ssh"`
}
func (p Package) Module(domain string) string {
return fmt.Sprintf("%s/%s", strings.TrimSuffix(domain, "/"), strings.ToLower(p.Name))
}
// Info gets Info from a vanity server
func (c *Client) Info(ctx context.Context) (Info, error) {
var info Info
resp, err := c.crud(ctx, Package{}, http.MethodOptions)
if err != nil {
return info, err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return info, fmt.Errorf("could not get info: %s", resp.Status)
}
if err := json.NewDecoder(resp.Body).Decode(&info); err != nil {
return info, err
}
return info, nil
}
// Add adds a new Package to a vanity server
func (c *Client) Add(ctx context.Context, pkg Package) error {
resp, err := c.crud(ctx, pkg, http.MethodPost)
if err != nil {
return err
}
if resp.StatusCode != http.StatusCreated {
return fmt.Errorf("could not add package: %s", resp.Status)
}
return nil
}
// Update updates a Package on a vanity server
func (c *Client) Update(ctx context.Context, pkg Package) error {
resp, err := c.crud(ctx, pkg, http.MethodPatch)
if err != nil {
return err
}
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("could not update package: %s", resp.Status)
}
return nil
}
// Remove removes a Package from a vanity server
func (c *Client) Remove(ctx context.Context, pkg Package) error {
resp, err := c.crud(ctx, pkg, http.MethodDelete)
if err != nil {
return err
}
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("could not remove package: %s", resp.Status)
}
return nil
}
func (c *Client) crud(ctx context.Context, pkg Package, method string) (*http.Response, error) {
payload, err := json.Marshal(pkg)
if err != nil {
return nil, err
}
req, err := c.newRequest(ctx, method, c.server, bytes.NewReader(payload))
if err != nil {
return nil, err
}
return c.http.Do(req)
}

45
go-vanity/source.go

@ -0,0 +1,45 @@
package vanity
import (
"errors"
"fmt"
"strings"
)
type SourceDirFile struct {
Dir string
File string
}
func GiteaSDF(pkg Package) SourceDirFile {
return SourceDirFile{
Dir: fmt.Sprintf("%s/src/branch/%s{/dir}", pkg.WebURL, pkg.Branch),
File: fmt.Sprintf("%s/src/branch/%s{/dir}/{file}#L{line}", pkg.WebURL, pkg.Branch),
}
}
func GitHubSDF(pkg Package) SourceDirFile {
return SourceDirFile{
Dir: fmt.Sprintf("%s/tree/%s{/dir}", pkg.WebURL, pkg.Branch),
File: fmt.Sprintf("%s/blob/%s{/dir}/{file}#L{line}", pkg.WebURL, pkg.Branch),
}
}
func GitLabSDF(pkg Package) SourceDirFile {
return SourceDirFile{
Dir: fmt.Sprintf("%s/-/tree/%s{/dir}", pkg.WebURL, pkg.Branch),
File: fmt.Sprintf("%s/-/blob/%s{/dir}/{file}#L{line}", pkg.WebURL, pkg.Branch),
}
}
func AnalyzeSDF(pkg Package) (SourceDirFile, error) {
switch {
case strings.Contains(pkg.WebURL, "gitea.com"):
return GiteaSDF(pkg), nil
case strings.Contains(pkg.WebURL, "github.com"):
return GitHubSDF(pkg), nil
case strings.Contains(pkg.WebURL, "gitlab.com"):
return GitLabSDF(pkg), nil
}
return SourceDirFile{}, errors.New("could not detect SDF")
}

194
go-vanity/vanity_test.go

@ -0,0 +1,194 @@
package vanity
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"os"
"strings"
"testing"
)
var (
server *httptest.Server
token = "TestingLibrary"
version = "VanityTest"
packages = []Package{
{
Name: "test1",
Description: "test1",
Branch: "main",
WebURL: "https://gitea.com/jolheiser/test1",
CloneHTTP: "https://gitea.com/jolheiser/test1.git",
CloneSSH: "https://gitea.com/jolheiser/test1",
},
}
)
func TestMain(m *testing.M) {
server = httptest.NewServer(http.HandlerFunc(testServer))
os.Exit(m.Run())
}
func TestClient(t *testing.T) {
ctx := context.Background()
client := New("", WithServer(server.URL))
// Info
checkInfo(t, client, 1)
pkg1 := Package{
Name: "test1",
Description: "test1",
Branch: "main",
WebURL: "https://gitea.com/jolheiser/test1",
CloneHTTP: "https://gitea.com/jolheiser/test1.git",
CloneSSH: "https://gitea.com/jolheiser/test1",
}
pkg2 := Package{
Name: "test2",
Description: "test2",
Branch: "main",
WebURL: "https://gitea.com/jolheiser/test2",
CloneHTTP: "https://gitea.com/jolheiser/test2.git",
CloneSSH: "https://gitea.com/jolheiser/test2",
}
// Add (without token)
if err := client.Add(ctx, pkg1); err == nil {
t.Log("adding without token should fail")
t.Fail()
}
// Add (with token)
client = New(token, WithServer(server.URL))
checkAdd(t, client, pkg1, pkg2)
// Info (after second package)
checkInfo(t, client, 2)
// Update package
checkUpdate(t, client, pkg1)
// Remove
checkRemove(t, client, pkg1)
// Info (final)
checkInfo(t, client, 1)
}
func checkInfo(t *testing.T, client *Client, numPackages int) {
info, err := client.Info(context.Background())
if err != nil {
t.Logf("info should not return error: %v\n", err)
t.Fail()
}
if info.Version != version || info.NumPackages != numPackages {
t.Log("info did not match expected")
t.Fail()
}
}
func checkAdd(t *testing.T, client *Client, pkg1, pkg2 Package) {
ctx := context.Background()
if err := client.Add(ctx, pkg2); err != nil {
t.Logf("pkg2 should be added: %v\n", err)
t.Fail()
}
// Duplicate package
if err := client.Add(ctx, pkg1); err == nil {
t.Log("pkg1 should already exist")
t.Fail()
}
}
func checkUpdate(t *testing.T, client *Client, pkg Package) {
ctx := context.Background()
// Update invalid package
if err := client.Update(ctx, Package{Name: "test4"}); err == nil {
t.Log("should not be able to update invalid package")
t.Fail()
}
// Update valid package
if err := client.Update(ctx, pkg); err != nil {
t.Logf("should be able to update valid package: %v\n", err)
t.Fail()
}
}
func checkRemove(t *testing.T, client *Client, pkg Package) {
ctx := context.Background()
if err := client.Remove(ctx, pkg); err != nil {
t.Logf("should be able to remove package: %v\n", err)
t.Fail()
}
// Remove (idempotent)
if err := client.Remove(ctx, pkg); err != nil {
t.Logf("should be able to remove package idempotently: %v\n", err)
t.Fail()
}
}
func testServer(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/":
switch r.Method {
case http.MethodOptions:
resp := Info{
Version: version,
NumPackages: len(packages),
}
_ = json.NewEncoder(w).Encode(resp)
case http.MethodPost, http.MethodPatch, http.MethodDelete:
if r.Header.Get(TokenHeader) != token {
w.WriteHeader(http.StatusUnauthorized)
return
}
var pkg Package
if err := json.NewDecoder(r.Body).Decode(&pkg); err != nil {
w.WriteHeader(http.StatusInternalServerError)
return
}
switch r.Method {
case http.MethodPost:
for _, p := range packages {
if p.Name == pkg.Name {
w.WriteHeader(http.StatusConflict)
return
}
}
packages = append(packages, pkg)
w.WriteHeader(http.StatusCreated)
case http.MethodPatch:
for idx, p := range packages {
if p.Name == pkg.Name {
packages[idx] = pkg
return
}
}
w.WriteHeader(http.StatusNotFound)
case http.MethodDelete:
for idx, p := range packages {
if p.Name == pkg.Name {
packages = append(packages[:idx], packages[idx+1:]...)
}
}
}
return
}
return
default:
name := strings.TrimPrefix(r.URL.Path, "/")
for _, pkg := range packages {
if pkg.Name == name {
_ = json.NewEncoder(w).Encode(pkg)
return
}
}
}
w.WriteHeader(http.StatusNotImplemented)
}

16
go.mod

@ -2,18 +2,18 @@ module go.jolheiser.com/vanity
go 1.16
replace go.jolheiser.com/vanity/go-vanity => ./go-vanity
require (
code.gitea.io/sdk/gitea v0.13.2
github.com/AlecAivazis/survey/v2 v2.2.8
github.com/cpuguy83/go-md2man/v2 v2.0.0 // indirect
github.com/go-chi/chi v4.1.2+incompatible
github.com/go-chi/cors v1.1.1 // indirect
github.com/google/go-github/v32 v32.1.0
github.com/pelletier/go-toml v1.8.1
github.com/urfave/cli/v2 v2.2.0
github.com/xanzy/go-gitlab v0.37.0
go.etcd.io/bbolt v1.3.5
go.jolheiser.com/beaver v1.0.2
go.jolheiser.com/overlay v0.0.2 // indirect
golang.org/x/crypto v0.0.0-20200820211705-5c72a883971a // indirect
golang.org/x/oauth2 v0.0.0-20200902213428-5d25da1a8d43
go.jolheiser.com/overlay v0.0.2
go.jolheiser.com/vanity/go-vanity v0.0.0-00010101000000-000000000000
golang.org/x/net v0.0.0-20200822124328-c89045814202 // indirect
golang.org/x/sys v0.0.0-20200909081042-eff7692f9009 // indirect
golang.org/x/text v0.3.3 // indirect
)

399
go.sum

@ -1,413 +1,62 @@
cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
cloud.google.com/go v0.38.0/go.mod h1:990N+gfupTy94rShfmMCWGDn0LpTmnzTp2qbd1dvSRU=
cloud.google.com/go v0.44.1/go.mod h1:iSa0KzasP4Uvy3f1mN/7PiObzGgflwredwwASm/v6AU=
cloud.google.com/go v0.44.2/go.mod h1:60680Gw3Yr4ikxnPRS/oxxkBccT6SA1yMk63TGekxKY=
cloud.google.com/go v0.45.1/go.mod h1:RpBamKRgapWJb87xiFSdk4g1CME7QZg3uwTez+TSTjc=
cloud.google.com/go v0.46.3/go.mod h1:a6bKKbmY7er1mI7TEI4lsAkts/mkhTSZK8w33B4RAg0=
cloud.google.com/go v0.50.0/go.mod h1:r9sluTvynVuxRIOHXQEHMFffphuXHOMZMycpNR5e6To=
cloud.google.com/go v0.52.0/go.mod h1:pXajvRH/6o3+F9jDHZWQ5PbGhn+o8w9qiu/CffaVdO4=
cloud.google.com/go v0.53.0/go.mod h1:fp/UouUEsRkN6ryDKNW/Upv/JBKnv6WDthjR6+vze6M=
cloud.google.com/go v0.54.0/go.mod h1:1rq2OEkV3YMf6n/9ZvGWI3GWw0VoqH/1x2nd8Is/bPc=
cloud.google.com/go v0.56.0/go.mod h1:jr7tqZxxKOVYizybht9+26Z/gUq7tiRzu+ACVAMbKVk=
cloud.google.com/go v0.57.0/go.mod h1:oXiQ6Rzq3RAkkY7N6t3TcE6jE+CIBBbA36lwQ1JyzZs=