Contents

Full Analysis to License Generation Process of Github Enterprise

😗Translated Content😗

This article is machine translated which hasn’t been proofreaded by the author. The info it contains may be inaccurate. The author will do his best to get back (when he has time) and revise these articles. 🥰

For Chinese version of this article, see here.

Overview

Github Enterprise is a code hosting and project collaboration platform tailored by Github for enterprise users. It provides two different deployment methods: public cloud and private cloud. This article will explain in detail the process of generating and verifying the license of Github Enterprise (hereinafter referred to as GHE) and the idea of reverse analysis. At the same time, I will share the one-click registration machine I wrote to kill it out of the box. I hope that some of the ideas shared in this article will also provide some help for everyone to quickly find breakthroughs in Linux system forensics, application reverse analysis, and authorization bypassing.

The interface of Github Enterprise is exactly the same as that of Github, and it contains almost all the features of Github (the missing features are currently only the Discussion in the project page), and additionally provides other advanced features that Github does not have. List a few that I am impressed with.

  1. SAML/CAS/LDAP SSO (without OAuth)
  2. Behavioral auditing (very fine-grained!)
  3. Access Authority control (including access rights for various functions such as Fork, Issue, PR, etc., project visibility, etc.)
  4. Code security scanning (Github Advanced Security, including code scanning, password private key scanning, and dependency library three blocks)
  5. Connect with Github (like Bing, you can search internal enterprise data on Github)

Through this article, readers can learn roughly as follows:

  1. The installation operation process of Github Enterprise (as for the function and use of GHE, it is basically the same as Github, you can try it yourself).
  2. Github Enterprise’s authorization verification mechanism, trust chain, verification mechanism implementation, (a not very strong) program obfuscation algorithm, a method to bypass the authorization verification mechanism without destroying the OTA function.
  3. How to reliably, future-proof crack a software product delivered as a * complete operating system *. How to do it: Package a dedicated Linux distribution ISO, bypass the target system disk, modify necessary files (such as authorization files) and inject a two-stage payload.
  4. Advanced usage of bash scripts: color output, multi-process/subprocess monitoring/Inter-process Communication/output capture, verifying the integrity of containers/images/in-image files/in-container files, etc.
  5. A completely open source Github Enterprise cracking tool!

Locate key information

First, we download the GHE installation package from the official website. As of the time of writing this article (2022.02), the latest version of the official website is 3.4.0.rc 1 (the registration machine tool shared later, after many tests, can guarantee stable operation on this version).

The official website provides three deployment methods, Hyper-V (VHD), OpenStack KVM (QCOW2), VMWare ESXi/vSphere (OVA). Here I choose the OVA format and deploy it to vSphere. Readers can also import OVA to VMWare Workstation or Virtualbox for trial, be sure to pay attention to resource requirements. When I deployed, I used the configuration of 2CPU, 16G RAM, 200G system disk + 150G data disk. The official recommended (< gtr = “29”/>) resource configuration is 4CPU, 32G RAM.

File System and Distribution Information

After deployment, we don’t rush into the system. Start the virtual machine with the Kali boot disk and check the partitions. The system disk is divided into two pieces, which we can use as A/B partitions. Since it is a Linux system, we need to know which distribution it is.

/2022/github-enterprise-reverse-engineering/assets/image-20220215122044-15updlk.png

Reading lsb_release source code shows that the distribution information is stored in the < gt r = “31”/> file. Reading this file reveals that the current system is Debian 10.

$ cat /media/kali/_/usr/lib/os-release
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
...

System Services and Key Directories

Since it is Debian, no matter what service it uses for task scheduling, it must be powered on through systemd. Let’s flip through < gt r = “32”/>, which will contain the automatic launch item of user configuration (or third-party configuration). It is easy to find that the large number of services starting with < gt r = “33”/> is very conspicuous, ghe is < gt r = “34”/> it ** h ** ub < gt r = “35”/> Abbreviation for nterprise. ( In the follow-up analysis, I also found that the service of nomad is also very important, which will be left to talk about later)

I originally thought that Github Enterprise should be referred to as GE for short, but I thought it was weird, but sure enough, the official acronym GHE looked more comfortable.

$ ls /etc/systemd/system/*.target.wants
/etc/systemd/system/getty.target.wants:
getty@tty1.service

/etc/systemd/system/github-enterprise.target.wants:
graphite-web.service

/etc/systemd/system/multi-user.target.wants:
chrony.service           enterprise-manage.service          grafana-server.service         open-vm-tools.service
console-setup.service    ghe-create-log-dirs.service        haproxy-cluster-proxy.service  rdnssd.service
consul.service           ghe-loading-page.service           haproxy-data-proxy.service     remote-fs.target
consul-template.service  ghe-reconfigure.service            haproxy.service                rsync.service
containerd.service       ghe-secrets.service                lo-enable-ipv6.service         ssh.service
cron.service             ghe-user-disk.service              networking.service             syslog-ng.service
dnsmasq.service          ghe-wait-for-certificates.service  nomad-jobs.service             sysstat.service
docker.service           github-enterprise.target           nomad.service                  ufw.service

/etc/systemd/system/network-online.target.wants:
networking.service

/etc/systemd/system/sockets.target.wants:
dm-event.socket  docker.socket

/etc/systemd/system/sysinit.target.wants:
blk-availability.service  keyboard-setup.service  lvm2-monitor.service  systemd-timesyncd.service
ghe-welcome.service       lvm2-lvmpolld.socket    resolvconf.service    virt-what.service

/etc/systemd/system/timers.target.wants:
apt-daily.timer  logrotate.timer  man-db.timer

Just click on a few services, and you can see that < gt r = “36”/> seems to be a key directory.

/2022/github-enterprise-reverse-engineering/assets/image-20220215123346-c80ouv7.png

What are these startup items for? We won’t see it until it’s booted up. However, in this directory, I found an interesting script < gt r = “38”/>. At a glance, we know that this script is equivalent to telling us directly that the ** authorization file is installed in ** < gt r = “39”/>, ** is a tar file set by gpg, and the compressed package contains a script called metadata.json. The file, which is the registration code information **.

#!/bin/bash
#/ Usage: ghe-license-info
set -e

GHE_LICENSE_FILE=${GHE_LICENSE_FILE:-/data/user/common/enterprise.ghl}

[ "$(whoami)" = "root" ] || [ -r "$GHE_LICENSE_FILE" ] || {
  exec sudo -u root GHE_LICENSE_FILE="$GHE_LICENSE_FILE" "$0" "$@"
  echo Run this script as the root user. >&2
  exit 1
}

if [ ! -f "$GHE_LICENSE_FILE" ]; then
  echo "No license file found ($GHE_LICENSE_FILE)." >&2
  exit 1
fi

info=$(gpg --skip-verify --no-permission-warning --decrypt "$GHE_LICENSE_FILE" 2>/dev/null |
  tar xO metadata.json |
  jq -c 'del(.customer_public_key,.customer_private_key)')

if [ "$1" == "-j" ]; then
  echo "$info"
else
  echo "$info" |
  jq -r 'to_entries | sort[] | "\(.key | tojson) : \(.value | tojson)"'
fi

Obfuscation algorithm

A full disk search for files with license names, except for the open source licenses of some open source components, these files are more prominent.

/data/enterprise-manage/current/lib/manage/models/license_settings.rb
/data/enterprise-manage/current/lib/manage/license.rb
/data/enterprise-manage/current/vendor/gems/ruby/2.7.0/gems/enterprise-crypto-0.4.22/lib/enterprise/crypto/license.rb
/data/enterprise-manage/current/vendor/gems/ruby/2.7.0/gems/enterprise-crypto-0.4.22/lib/enterprise/crypto/license_vault.rb

But when you open it, you will find that these files are all confused

__ruby_concealer__ "x\x9C]\x9A\xFB[\xDAz\xB3\xF6\xBF9\x90\x84\x9CH
...
+\f\xC1\xF8\xF3\xDD\x84\x82\"\xD9\xF9\x1D\xC00\xA1\xB7mGMp7~|\xD9\xFF\a\xBD\xE1Yb"

Unpacking these files is also easy. After using zlib inflate, use the following string as the Key XOR obfuscated string. Note that there is a space at the end.

"This obfuscation is intended to discourage GitHub Enterprise customers from making modifications to the VM. We know this 'encryption' is easily broken. "

Obfuscation functions are packaged in ruby. No analysis here.

$ strings /usr/local/bin/ruby | grep encryption
This obfuscation is intended to discourage GitHub Enterprise customers from making modifications to the VM. We know this 'encryption' is easily broken.

Decrypt using the following script.

require 'zlib'

def decrypt(s)
    i, plaintext = 0, ''
    key = "This obfuscation is intended to discourage GitHub Enterprise customers from making modifications to the VM. We know this 'encryption' is easily broken. "

    Zlib::Inflate.inflate(s).each_byte do |c|
        plaintext << (c ^ key[i%key.length].ord).chr
        i += 1
    end
    plaintext
end

content = File.open(ARGV[0], "r").read
content.sub! %Q(__ruby_concealer__), " decrypt "
plaintext = eval content

puts plaintext

The corresponding encryption script. In fact, we don’t need the encryption script, and it’s no problem to put it directly after the code is changed without encryption. It’s just Tuyile.

require 'zlib'

def encrypt(s)
    i, ciphertext = 0, ''
    key = "This obfuscation is intended to discourage GitHub Enterprise customers from making modifications to the VM. We know this 'encryption' is easily broken. "

    s.each_byte do |c|
        ciphertext << (c ^ key[i%key.length].ord).chr
        i += 1
    end
    Zlib::Deflate.deflate(ciphertext)
end

content = File.open(ARGV[0], "r").read
ciphertext = encrypt(content)
puts "__ruby_concealer__ #{ciphertext.inspect}"

Save this file as /tmp/dec.rb and use the following bash command to dump all encrypted programs in the GHE system partition (which can be chrooted from Kali Live).

cd /data/enterprise-manage/current
find lib/ -name '*.rb' | xargs -I% sh -c ' mkdir -p /tmp/$(dirname %) && grep -o  __ruby_concealer__ % && ruby /tmp/dec.rb % > /tmp/% || cp % /tmp/%'^C

List of Obfuscation Protected Files

Just search for the .rb file and check that the file content contains < gt r = “40”/>. The full list is too long.

Authorization file structure

We have not yet registered the code, the system has not started yet, it has already obtained so much information. According to international practice, we can not a whole registration code?

Just fool a mailbox to register from the Github Enterprise official website, and then you can apply for a 45-day trial license. Through the file decompression method learned earlier, we can take a look at the content inside.

Install GnuPG (windows can install gpg4win) and use gpg to decrypt the ghl authorization file obtained from the official website before. Said to be decrypted, in fact, no password is required. After decryption, use 7-Zip to decompress tar to get the files inside. It can be seen that in addition to metadata.json, it also contains two sets of keys.

gpg --skip-verify --no-permission-warning --decrypt github-enterprise-195560.ghl.tar github-enterprise-195560.ghl
github-enterprise-195560.tar
   metadata.json
├───customer
       pubring.gpg
       secring.gpg
└───license
        pubring.gpg
        secring.gpg

The fingerprints of the two keys are exactly the same, and the contents of the < gt r = “41”/> and < gt r = “42”/> fields in the metadata.json are also exactly the same as the two keys.

/2022/github-enterprise-reverse-engineering/assets/image-20220221135134-uvji73d.png

After removing the above two fields, the content of the metadata.json is as follows. < gt r = “44”/> in the following file is the business name I filled in,

{
    "reference_number": "13eb75",
    "company": "Company",
    "ssh_allowed": true,
    "cluster_support": false,
    "croquet_support": false,
    "custom_terms": false,
    "support_key": null,
    "unlimited_seating": true,
    "perpetual": false,
    "evaluation": true,
    "learning_lab_seats": 0,
    "learning_lab_evaluation_expires": null,
    "insights_enabled": false,
    "insights_expire_at": null,
    "advanced_security_enabled": false,
    "advanced_security_seats": 0,
    "seats": 0,
    "expire_at": "2022-03-31T23:59:59-07:00"
}

Wouldn’t it be that I changed the content of this JSON and put it in a package with GPG?

Yes, and no.

Authorization Verification Algorithm

As we all know, GPG is used to verify digital signatures, just like the CA certificate system, a complete chain of trust needs to be built. But the CA is centralized, just like the Authority in its name, all users rely on the list of CAs printed in the operating system from the factory to verify the authenticity of each other.

GPG relies on everyone to trust each other. A signs B’s certificate, and you can trust all the data signed by B’s certificate. But A must check by himself whether B’s certificate was sent through a trusted channel.

And Github’s approach is to hardcode the B certificate in its own code, isn’t it safe?

What a genius!

Then when we generate the authorization file, we need to add a digital signature signed by a trusted key to the authorization file. Of course, only one public key will be stored in the system. In order to generate the signature, we need to generate a private key ourselves and replace the corresponding public key into the system to take over the entire chain of trust.

Smart readers, you must already have ideas here. From here, I will focus on the results and some key information after sorting out, and the procedural description will be ignored as much as possible.

Source code and library namespace mapping

类名 用途 ruby require File location
Enterprise::Crypto 授权文件与 OTA 包读写、验证 enterprise/crypto /data/enterprise-manage/current/vendor/gems/ruby/2.7.0/gems/enterprise-crypto-0.4.22/lib/enterprise/crypto.rb
Enterprise::Crypto::Vault 装 key 的容器,每个 key 一个容器,都经过 master key 校验的子 key enterprise/crypto /data/enterprise-manage/current/vendor/gems/ruby/2.7.0/gems/enterprise-crypto-0.4.22/lib/enterprise/crypto/vault.rb
Manage 后台主程序,里面初始化了授权模块 manage /data/enterprise-manage/current/lib/manage.rb
Manage::License 后台程序里负责授权的模块 manage/license /data/enterprise-manage/current/lib/manage/license.rb

The Manage module is the Admin console, which is forwarded to nginx to unicorn’s daemon via haproxy, accessed from port 8443. This program is a backend program when GHE is installed.

The Crypto module is a public module, and all signature verification, file reading and writing (including the decompression of authorized files and the decompression of OTA packages) are carried out by this module.

Vault is a key class in the module. This class is responsible for the storage of public and private keys, the signature of root keys and the verification of fingerprints. Its three subclasses LicenseVault, CustomerVault, and PackageVault, as the name suggests, are responsible for specific functions.

Key key

The root key hardcoding for signature verification is in the following file.

/data/enterprise-manage/current/vendor/gems/ruby/2.7.0/gems/enterprise-crypto-0.4.22/lib/enterprise/crypto/vault.rb

The fingerprint of the root key is < gt r = “45 “/> ,** this fingerprint is the root of trust for GHE **.

pub   rsa4096/1AF590C895016367 2011-08-06 [SC]
      E422F81088AE9ABD5A6CBF7D1AF590C895016367
uid                 [ unknown] GitHub Master Key <support@github.com>
sig 3        1AF590C895016367 2011-08-06  GitHub Master Key <support@github.com>
sub   rsa4096/4ED44822D1ED9253 2011-08-06 [E]
sig          1AF590C895016367 2011-08-06  GitHub Master Key <support@github.com>

The root key signs 3 secondary keys. Package is the signature upgrade package, license is the signature

These three paths are referenced in < gt r = “46”/>.

/data/enterprise/package.gpg
/data/enterprise/license.gpg
/data/enterprise/customer.gpg

As can be seen from the output of gpg, all three keys contain the signature of the Master Key.

pub   rsa4096/8BDF429EF29F9D7E 2011-08-23 [SC]
      74708F1DFE79BF5FE2DDEC458BDF429EF29F9D7E
uid                 [ unknown] GitHub Customer Key <support@github.com>
sig 3        8BDF429EF29F9D7E 2011-08-23  GitHub Customer Key <support@github.com>
sig          1AF590C895016367 2011-08-23  GitHub Master Key <support@github.com>
sub   rsa4096/FC78CDE72E0420D4 2011-08-23 [E]
sig          8BDF429EF29F9D7E 2011-08-23  GitHub Customer Key <support@github.com>

pub   rsa4096/1E9E2B9CD80BFEB7 2011-08-23 [SC]
      C49B7FA81A0BC6D26DEAC2BA1E9E2B9CD80BFEB7
uid                 [ unknown] GitHub License Key <support@github.com>
sig 3        1E9E2B9CD80BFEB7 2011-08-23  GitHub License Key <support@github.com>
sig          1AF590C895016367 2011-08-23  GitHub Master Key <support@github.com>
sub   rsa4096/4AAC02EA164A2313 2011-08-23 [E]
sig          1E9E2B9CD80BFEB7 2011-08-23  GitHub License Key <support@github.com>

pub   rsa4096/B2B935E31729A28E 2011-08-18 [SC]
      663BE6981E181CC121BC4CADB2B935E31729A28E
uid                 [ unknown] GitHub Package Key <support@github.com>
sig 3        B2B935E31729A28E 2011-08-18  GitHub Package Key <support@github.com>
sig          1AF590C895016367 2011-08-18  GitHub Master Key <support@github.com>
sub   rsa4096/47443DAD9E39C3C3 2011-08-18 [E]
sig          B2B935E31729A28E 2011-08-18  GitHub Package Key <support@github.com>

The public and private key pairs contained in the authorization file are signed by Customer Key.

/2022/github-enterprise-reverse-engineering/assets/image-20220221152819-j9ujt25.png

Overall verification process

The overall verification process is shown below. Take the initialization process of the Admin console as an example.

  1. M anage.setup initialize the three Vaults and read the contents of the original file (all parameters of the new function as an array) into the < gt r = “48”/> inside the corresponding Vault.

    Enterprise::Crypto.license_vault  = LicenseVault.new(license_key)
    Enterprise::Crypto.customer_vault = CustomerVault.new(customer_key)
    Enterprise::Crypto.package_vault  = PackageVault.new(package_key)
    
  2. When the < gt r = “50”/> context is initialized, read < gt r = “51”/> and verify the subkey.

    1. Enterprise::Crypto.with_vault-> Enterprise::Crypto::Vault.open!

      1. < Gt r = “54”/> Load ** hardcoding master public key blob into GPG **

      2. Enterprise::Crypto::Vault.load_vault

        1. Load the contents of < gt r = “55”/> into the GPG one by one, and make sure they have the same fingerprint, save the < gt r = “56”/> variable
        2. If the private key is imported (queried via the < gt r = “57”/> variable), make sure the private key is signed the same as the public key < gt r = “58”/>
        3. If the public key is imported (queried via the < gt r = “59”/> variable), ** make sure the public key is signed by the master key ** < gt r = “60”/>
      3. < Gt r = “61”/> Repeat 2.1.2 to verify the key again.

        1. < Gt r = “62”/> Make sure the master pubkey is the same as ** hardcoding’s fingerprint **.
        2. < Gt r = “63”/> If the public key was imported in step 2.1.2, ** make sure the public key is signed by the master key **
        3. < Gt r = “64”/> If the private key was imported in step 2.1.2, make sure the private key is signed the same as the public key
  3. Perform authorization verification, such as License loading or OTA package decompression. Both of the following examples call < gt r = “65”/>

    # Manage::License
          @crypto = begin
            Enterprise::Crypto::License.load(data)
                    rescue Enterprise::Crypto::Error
                      nil
          end
    # Enterprise::Crypto::Package
          def self.extract(package_path, output_path, vault = Crypto.package_vault)
            tar_file = Crypto.with_vault(vault) do
              vault.read_package(File.open(package_path))
            end
    
            from_tar(tar_file.path, output_path)
    
            verify_extraction(output_path)
    

The bold mark above is our breaking point. In fact, modifying two functions in the vault.rb is enough.

/2022/github-enterprise-reverse-engineering/assets/image-20220221160434-iy6x3s3.png

It is not safe to have all validations return True. If you directly modify < gt r = “67”/>, this will completely invalidate the online upgrade function, and the downloaded upgrade package will not pass the verification. Inserting only fingerprints does not need to replace the complete master pubkey data, reducing the amount of patches that need to be changed to facilitate the generation of patches. This modification method is the most concise and effective.

However, if we modify it in this way, we need to fill in the modified vault.rb with the Master Key fingerprint and SubkeyID generated by ourselves. Therefore, we also need to convert the key into a key and then package all the patches.

Generate authorization file

Save the following two scripts as keygen.sh, keygen.rb, and copy them into the GHE chroot directory together with the modified vault.rb in the previous step.

Among them, the first script is used to generate the GPG key, and the second script is used to generate the authorization file.

#!/bin/bash
# keygen.sh

script_path=$(cd -P -- "$(dirname -- "$0")" && pwd -P)
key_path="./keys"

GNUPGHOME="$script_path/gpg"

mkdir -p "$GNUPGHOME" && chmod go-rwx "$GNUPGHOME"
mkdir -p "$key_path"

makekey () {
  echo "Creating key for $@" >&2
  gpg --batch --gen-key 2> >(perl -n -e '/\/([A-F0-9]{40}).rev/ && print $1') <<EOF
%no-protection
Key-Type: RSA
Key-Length: 4096
Subkey-Type: RSA
Subkey-Length: 4096
Name-Real: $1
Name-Email: $2
Expire-Date: 0
EOF
}

signkey () {
  echo "Signing $2 with signature of $1"
  printf 'y\nsave\n' | gpg --command-fd 0 --status-fd 2 --local-user "$1" --edit-key "$2" sign
}

exportkey () {
  gpg --export --armor "$1" > "$key_path/$2.gpg"
}

exportseckey () {
  gpg --export-secret-keys --armor "$1" > "$key_path/$2.gpg"
}

mkey=$(makekey "Github Master Key" "test@github.com")
lkey=$(makekey "Github License Key" "test@github.com")
ckey=$(makekey "Github Customer Key" "test@github.com")


set -x
signkey "$mkey" "$lkey"
signkey "$mkey" "$ckey"


exportkey	"$mkey" master
exportseckey	"$mkey" master_sec
exportkey	"$lkey" license
exportseckey	"$lkey" license_sec
exportkey	"$ckey" customer
exportseckey	"$ckey" customer_sec

keygen.rb call the GHE internal module to generate the authorization file.

# keygen.rb
# frozen_string_literal: true
require "enterprise/crypto"


def main()
  license = generate_example_license(0, Time.now + days, license_metadata)
  File.open(ENV["LICENSE_FILENAME"] || "license.ghl", "wb") { |f| f << license.to_bin }
end

def generate_example_license(seats, expire_at, metadata = {})
  customer_sec_key = File.read("keys/customer_sec.gpg")
  customer_pub_key = File.read("keys/customer.gpg")
  license_sec_key  = File.read("keys/license_sec.gpg")
  license_pub_key  = File.read("keys/license.gpg")

  name  = ENV["LICENSE_NAME"] || "Test User"
  email = ENV["LICENSE_EMAIL"] || "test@example.com"

  Enterprise::Crypto.customer_vault =
    Enterprise::Crypto::CustomerVault.new(customer_sec_key, customer_pub_key, blank_password: true)

  Enterprise::Crypto.license_vault =
    Enterprise::Crypto::LicenseVault.new(license_sec_key, license_pub_key, blank_password: true)

  customer = Enterprise::Crypto::Customer.generate(name, email)
  license = customer.generate_license(seats, expire_at, support_key, metadata)
end

def license_metadata
  company            = ENV["LICENSE_COMPANY"] || "Baidu Inc."
  ref_number         = ENV["LICENSE_KEY_REFERENCE_NUMBER"] || "0b634b"
  license_ref_number = ENV["LICENSE_REFERENCE_NUMBER"] || "2df6d8"
  order_ref_number   = ENV["LICENSE_ORDER"] || "81e4b7"

  {
    reference_number: ref_number,
    order: order_ref_number,
    license: license_ref_number,
    company: company,
    ssh_allowed: true,
    support_key: support_key,
    cluster_support: true,
    advanced_security_enabled: true,
    unlimited_seating: true,
    perpetual: true
  }
end

def seats
  (ENV["LICENSE_SEATS"] || 100).to_i
end

def days
  60 * 60 * 24 * (ENV["LICENSE_DAYS"] || 365 * 10).to_i
end

if __FILE__ == $0
  main()
end

Don’t rush to run. After copying the file, remember to mount the /dev. When generating the key, you need to call the random number generator in this directory. Enter the chroot shell to run keygen.sh

GHE_ROOT=/media/root/_

cp keygen.sh keygen.rb "$GHE_ROOT/root/"
mount /dev $GHE_ROOT/dev
chroot $GHE_ROOT bash

# 在 chroot shell 内
export SHELL=/bin/bash PATH="/usr/share/rbenv/shims:$PATH" RBENV_VERSION=$(cat /etc/github/ruby_version) GEM_PATH=/data/enterprise-manage/current/vendor/gems/ruby/2.7.0
cd /root/
bash keygen.sh

After running keygen.sh, the keys subdirectory in the current directory will contain the master public and private keys and two pairs of subkeys, customer and license.

Use the gpg command to get the Master Key fingerprint and SubkeyID. Taking the pair of keys I generated as an example, the fingerprint is < gt r = “68”/>, and the SubkeyID is < gt r = “69”/>.

$ ls keys/
customer.gpg
customer_sec.gpg
license.gpg
license_sec.gpg
master.gpg
master_sec.gpg

$ gpg --with-fingerprint --keyid-format LONG --import-options show-only --import ./keys/master.gpg
pub   rsa4096/985269A0B58C25A5 2022-02-17 [SCEA]
      Key fingerprint = 6805 C30E 66B0 0B08 5D24  D9EC 9852 69A0 B58C 25A5
uid                            Github Master Key <security@github.com>
sub   rsa4096/D633FCD383735413 2022-02-17 [SEA]

Fill these two data into the modified vault.rb and copy it to the location of the original file (patch it to pass the verification). Then fill in the appropriate parameters through environment variables and run keygen.rb.

It should be noted that the value of < gt r = “70”/> will be used as the enterprise name displayed on the GHE interface, which cannot be modified after the authorization file is created.

ruby enc.rb vault.rb > vault.enc.rb  # optional
cp vault.enc.rb /data/enterprise-manage/current/vendor/gems/ruby/2.7.0/gems/enterprise-crypto-0.4.22/lib/enterprise/crypto/vault.rb

LICENSE_NAME="Known Rabbit" \
LICENSE_EMAIL="sales@github.com" \
LICENSE_COMPANY="KR" \
LICENSE_KEY_REFERENCE_NUMBER=$(dd if=/dev/urandom bs=1 count=3 | xxd -ps) \
LICENSE_REFERENCE_NUMBER=$(dd if=/dev/urandom bs=1 count=3 | xxd -ps) \
LICENSE_ORDER=$(dd if=/dev/urandom bs=1 count=3 | xxd -ps) \
ruby keygen.rb

After running, a license.ghl will be generated in the directory, because we are completely calling the authorization file created by GHE’s own code, so as long as the file is created successfully, the file must be verified as a valid authorization file by GHE. Copy this file and you can use it.

Files that need to be overwritten

In general, we need to overwrite our modified vault.rb to all locations where this file was called.

Due to the complexity of the overall architecture of GHE, the locations we covered before can only ensure that we can smoothly pass the initial setup of GHE (Admin console service on port 8443). Most of GHE’s services are located in Docker containers, and Nomad is responsible for the lifecycle management of these services.

File location in rootfs:

# Patch Management Console
/data/enterprise-manage/current/vendor/gems/ruby/2.7.0/gems/enterprise-crypto-0.4.22/lib/enterprise/crypto/vault.rb

Among them, the containers that verify the authorization information include < gt r = “71”/> (which is used to run database structure migration when GHE is initialized) and < gt r = “72”/> (GHE foreground web service), which RANDOM_UUID randomly generated every time When the service starts, once the program in it is closed (< gt r = “73”/>), it will be automatically deleted by nomad (< gt r = “74”/>) and a new container will be created. So you must overwrite the file in the image for cracking. In the latest version 3.4.0.rc 1, the container image is named < gt r = “75”/>. The vault file location inside the container:

/github/vendor/gems/2.7.1/ruby/2.7.0/gems/enterprise-crypto-0.4.23/lib/enterprise/crypto/vault.rb

Every time the virtual machine starts, or the system settings are modified through the Admin console, nomad will call < gt r = “76”/>, and load the mirror backup located in the following location by the method < gt r = “77”/>, logged at < gt r = “78”/>

/data/docker-images/github:4780f0371bdde4e035b372fdd8f373c11a3ef9c2#sha256:c837ca7816c67152409c4f5446d0061e9d5b6329d6cd0d62bc504c0df416adc7.tar

Therefore, either delete this image backup file so that the system cannot overwrite our patched image. Or patch this image backup as well. There are many details involved in patch image backup, so I won’t repeat them again. For the specific method, please refer to the installation script in my registration machine.

Packaging and Encapsulation of the Keygen - “Cracking a Dedicated Operating System”

The biggest difference between cracking GHE and other software is that the software itself is distributed as a complete operating system.

To crack a PE file, we only need to change a few assembly statements. To crack a Docker container, we just need to bind mount the patch file into it. But what about cracking an operating system? If we want to crack the operating system before it starts-up, we can only beat the magic with magic - another operating system to crack it.

Now test everyone, what is the smallest Linux distribution?

Linux Arch?

No, no, that thing is too big, the Live CD is more than 600 megabytes, and the smallest is 480M after I have reduced it. Alpine Linux may still rely on the point spectrum, but we usually use it as the bottom package of the container, and there are more than 100 megabytes packed with the core.

Now we ask Slitaz Linux to sit on the podium. Slitaz Linux that would be awesome. The entire rootfs was 29M before compression, plus Linux cores compressed together, the output ISO is only 16M!

/2022/github-enterprise-reverse-engineering/assets/image-20220219125952-cxemkmf.png

This is the base image suitable for making our “cracking dedicated operating system”. We named it ghec-os (** G ** it ** h ** ub < gt r = “80”/> nterprise < gt r = “81”/> rack < gt r = “82”/>)

Slitaz Linux provides an ISO customization tool called Tazlito. In short, we only need to use it to generate a template (flavor), fill in the template with the additional files we added according to the directory structure, and fill in the configuration file. The overall process described in the official document is as follows

Tazlito automates the process of building a customized ISO, and the method is quite straight-forward:

  1. find a template (flavor) to work on ($ tazlito list-flavors)
  2. download the template ($ tazlito get-flavor flavor_name)
  3. look at the flavor’s packages ($ tazlito show-flavor flavor_name)
  4. extract the flavor ($ tazlito extract-flavor flavor_name)
  5. put in a rootfs folder (make one by copying in files from /etc/ to /home/slitaz/distro )
  6. add additional files using the folder /home/slitaz/distro/addfiles (optional)
  7. change options ($ tazlito configure)
  8. create the ISO image ($ tazlito gen-distro)
  9. burn to CD or USB stick ($ tazusb gen-iso2usb)

The process of packaging ghec-os is also a long process of encountering various problems and solving them.

In short, in ghec-os, I added a program (crack.sh) that modifies the GHE root filesystem on the basis of the official base flavor, this program replaces the root key, the intermediate key and adds another program (crack-step2.sh) Inject the GHE startup item, detect the container after Nomad starts the Docker container, and crack the container image and container image backup as needed, and display the real-time status on the VM end point (uncracked, partially cracked, cracking running, cracking successful pending Restart, cracked).

In addition, I also changed the default user password to root: root, modified the motd login prompt, and added a soft link in the /usr/bin directory, so that after logging in to the root user, I can directly run < gt r = “83”/> to complete the crack.

/2022/github-enterprise-reverse-engineering/assets/image-20220222234059-65zvlfg.png

Summary of some problems encountered in the packaging process:

  1. The package manager of the Slitaz system installs the source domain name, which cannot be resolved in 114dns. It can be solved by replacing it with Ali DNS
  2. A package named linux-scsi must be added when packaging. The SCSI controller used by VMWare requires the driver in this package, otherwise the disk will not be recognized.
  3. When running < gt r = “85”/>, you must add environment variables named < gt r = “86”/>, which is a bug of tazlito, otherwise the package list in the distro-packages.list cannot be recognized properly.
  4. distro-packages.list this file cannot contain blank lines, otherwise it will flash when building the ISO. Packages with a minus sign in their name cannot be at the end of the list and need to be inserted in the middle.

In the process of writing the cracking shell script, I also encountered a lot of challenges. Friends who have the following questions like me can extract the programs in the registration machine by themselves and see how I implement the following requirements.

  1. How to display colored text in shell when there is no bash and no tput?
  2. How to verify the integrity of a file? How to verify the integrity of a single file in a container? How to verify the integrity of a single file in an image? How to verify the integrity of an image as a whole?
  3. How to implement multi-process with Bash? How to implement state monitoring of child processes? How to pass messages between processes and functions? How to maintain program state, capture output and child process status codes like systemd?
  4. How to ensure a stable program state? For example, against the update of the target program, it will not crash after cracking, and ensure the consistency of the results when the cracking program is run multiple times (avoid backup being overwritten, double patching, etc.).

The registration machine is done, let’s briefly talk about how to use it.

Use of the registration machine

After importing the OVA of GHE according to the official tutorial, set the CDROM as the priority startup item in the virtual machine BIOS, enter the Slitaz Linux startup interface, and enter the system once.

/2022/github-enterprise-reverse-engineering/assets/image-20220223222231-yhexxi6.png

After logging in, run the command < gt r = “89”/>. The user will be prompted to confirm before the first operation, just enter y to enter.

/2022/github-enterprise-reverse-engineering/assets/image-20220223222416-bxi858n.png

Run until the last prompt Crack module loaded. In the virtual machine settings, cancel the option of the optical drive “connect at startup”. Then restart the virtual machine.

/2022/github-enterprise-reverse-engineering/assets/image-20220223222615-kpwwq6w.png

After reboot, access the GHE IP, upload the license and complete the first initialization configuration. ** It is strongly recommended to add an SSH Key ** when initializing the configuration, so that you can go in and see the log when there is an error later.

/2022/github-enterprise-reverse-engineering/assets/image-20220223223202-szi69dv.png

After completing the initialization configuration, you will enter the waiting interface on the right side of the figure below. At this time, you can notice that the occupied space displayed in Storage will become larger and larger. At this time, GHE is decompressing the docker image. When the space grows to around 13G, GHE will start all docker containers to start database structure migration. The registrar starts cracking the docker image at this time, and displays a line of blue words indicating the running status and the PID of the child process.

When the crack runs, select “Restart Client” in VMWare, don’t click Force Restart. Running migrations will run on the right at this time. Don’t wait for it to finish running, it will report an error if you don’t restart it.

/2022/github-enterprise-reverse-engineering/assets/image-20220223225233-0ehxfg8.png

After restarting, GHE will continue to run the configuration process. Just wait. If all goes well, the configuration page and end point are all green.

/2022/github-enterprise-reverse-engineering/assets/image-20220223225514-4u672yh.png

Then click Visit your instance to see the main interface!

/2022/github-enterprise-reverse-engineering/assets/image-20220223231149-bcpuc73.png

About getting the registration machine: Follow the official account Rabbit Unknown and reply in the background: < gt r = “96”/> Get the download address. md5sum:

8c053c113054c4d2e089b2192e393147  ghe-crack.iso

Conclusion

GHE is a product full of ambition, from simple Git hosting at the beginning, to CI/CD (Actions), Package Repo, Discussion, Security Advisory, and now a full-featured on-premise version. In the process of trying out GHE, from a product perspective, Microsoft must be committed to making it a strong competitor to Gitlab EE. Personally, I think Github Actions is a whole era ahead of Gitlab CI/CD Pipelines.

Interestingly, the main framework of Github Enterprise is written in ruby, but the Github Actions part is written in C #plus Powershell. It seems that it is indeed the product of Microsoft’s acquisition.

While enjoying all the convenience brought by Github, objectively speaking, I also found that GHE also has many, many problems.

  1. The whole system is very unfriendly to the operation and maintenance personnel.

    1. It is very inconvenient to read the logs. The services all run in Docker, but the logs are not in Docker logs, and are scattered in various directories. There is no official documentation.
    2. There is no way to configure the switches and instance scaling of each component. For example, even if the Actions function is not used, the docker service in the background will not be turned off.
    3. With Docker, service scheduling is not the common compose, swarm, k8s, but nomad.
  2. The resource usage ** is extremely exaggerated **, the boot occupies 15G of memory, 70G of hard disk, and the disk write volume is 10G after 12 hours of operation - this is still the case without a single project in the instance.

  3. Unbearable little bug.

    1. The database does not seem to have time zone support on, and accessing all time-related settings will cause HTTP 500 crashes (timed reminders, time-limited personal status, etc.). Why is it not suitable for foreigners?
    2. The Github Actions and Packages functions require S3 Storage, but analyzing the Powershell script in the container found that a region code < gt r = “97”/> was hardcoding in the deployment service, resulting in an infinite loop of the Action deployment process. Why don’t foreigners use it?
    3. Modifying any configuration requires restarting all services, and the restart time is calculated in hours. The longest one took 12 hours. This down time is afraid that the operation and maintenance calls will be blown up.
  4. The back end is sewn up by Ruby and C\ #. Although the front end seems to have a high degree of integration, running MySQL + MSSQL two databases directly leads to a surge in memory occupancy, and the MSSQL service will gradually fill up the memory (16G) over time. Really have your MS.

  5. Supports internal enterprise CA (certificate manual import), but does not support private ACME services within the enterprise. Modifying the ACME server address of hardcoding in the script can be done, but isn’t LetsEncrypt used all over the world?

The last thing I want to say is that for personal use, unless there is a computer room at home, the productivity improvement benefits brought by the function itself are far less than the cost of computing/storage resources (including electricity bills) consumption **. I share this registration machine with you, and I also hope that through learning (bai) and learning (piao) it together, I will have the opportunity to mention bugs to Microsoft. Maybe it can be optimized and optimized!