Chef - zero (chef-zero and knife-zero)
24 Jan 2016list of commands required to prepare and cook umka node.
http://www.creationline.com/en/lab/8323:
knife-zero starts chef-zero server locally (it’s used to store node policies), logs into remote node via ssh and runs chef-client there using policies from local chef-zero server (chef-client accesses chef-zero server with SSH port forwarding - it thinks that chef-zero server is installed locally on the node)
https://www.coveros.com/knife-zero/:
This plugin creates an ssh tunnel so that when chef-zero listens on the master node, and the remote node tries to http connect to itself, it actually tunnels back to the listening master node.
the reason why remote node “tries to http connect to itself”:
/etc/chef/client.rb:
chef_server_url "chefzero://localhost:8889"
remote node
users:
devops
- used by Chef-
deploy
- used by Capistrano -
create
devops
user:# ssh root@<remote_node_ip> / enter root password # useradd devops -m -s /bin/bash -G sudo # passwd devops Enter new UNIX password: devops Retype new UNIX password: devops passwd: password updated successfully
local ws
add new host to SSH config:
~/.ssh/config:
Host tap349
User devops
Hostname DIGITAL_OCEAN_IP
IdentityFile ~/.ssh/home/id_rsa
ForwardAgent yes
ServerAliveInterval 180
add your public key to authorized keys for devops user on remote node
$ ssh tap349 'mkdir -p ~/.ssh'
/ enter devops password
$ cat ~/.ssh/home/id_rsa.pub | ssh tap349 "cat >> ~/.ssh/authorized_keys"
/ enter devops password
now login with ssh tap349
should work without password.
install chefdk
$ brew cask update
$ brew cask install chefdk
NOTE: Berkshelf is included as part ChefDK
update chef
inside chefdk
since knife-zero
needs chef
~> 12.6.0
$ gem install appbundle-updater
$ rvmsudo appbundle-updater chefdk chef master
$ rvmsudo appbundle-updater chefdk berkshelf master
$ chef --version
Chef Development Kit Version: 0.10.0
chef-client version: 12.6.0
berks version: 4.0.1
kitchen version: 1.4.2
install knife-zero
chef plugin:
- http://www.creationline.com/en/lab/8323
- https://github.com/higanworks/knife-zero
- https://knife-zero.github.io/20_getting_started
- https://www.coveros.com/knife-zero/
$ chef gem install knife-zero
generate chef repo
repo is used to orchestrate provisioning target nodes, it stores:
- Berksfile
- environments
- data_bags
- node files themselves
Berksfile is used to specify cookbook dependencies:
- community cookbooks from Chef supermarket
- wrapper or environment cookbooks (with either
path
orgit
option)
community cookbooks should not be vendored inside Chef repo -
they should be strored in ~/.bershelf/cookbooks/ (this is default
when running berks install
)
$ chef generate repo umka
$ tree -aF -L 3 -I .git umka
umka
├── .chef-repo.txt
├── .gitignore
├── LICENSE
├── README.md
├── chefignore
├── cookbooks/
│ ├── README.md
│ └── example/
│ ├── README.md
│ ├── attributes/
│ ├── metadata.rb
│ └── recipes/
├── data_bags/
│ ├── README.md
│ └── example/
│ └── example_item.json
├── environments/
│ ├── README.md
│ └── example.json
└── roles/
├── README.md
└── example.json
bootstrap remote node
$ knife zero bootstrap tap349 --sudo
WARNING: No knife configuration file found
Doing old-style registration with the validation key at ...
Delete your validation key in order to use your user credentials instead
Connecting to tap349
tap349 knife sudo password:
Enter your password:
tap349
tap349 -----> Installing Chef Omnibus (-v 12)
tap349 downloading https://omnitruck-direct.chef.io/chef/install.sh
tap349 to file /tmp/install.sh.4084/install.sh
tap349 trying wget...
tap349 Getting information for chef stable 12 for ubuntu...
tap349 downloading https://omnitruck-direct.chef.io/stable/chef/metadata?v=12&p=ubuntu&pv=14.04&m=x86_64
tap349 to file /tmp/install.sh.4088/metadata.txt
tap349 trying wget...
tap349 url https://opscode-omnibus-packages.s3.amazonaws.com/ubuntu/14.04/x86_64/chef_12.6.0-1_amd64.deb
tap349 md5 5cfc19d5a036b3f7860716bc9795a85e
tap349 sha256 e0b42748daf55b5dab815a8ace1de06385db98e29a27ca916cb44f375ef65453
tap349 version 12.6.0downloaded metadata file looks valid...
tap349 downloading https://opscode-omnibus-packages.s3.amazonaws.com/ubuntu/14.04/x86_64/chef_12.6.0-1_amd64.deb
tap349 to file /tmp/install.sh.4088/chef_12.6.0-1_amd64.deb
tap349 trying wget...
tap349 Comparing checksum with sha256sum...
tap349 Installing chef 12
tap349 installing with dpkg...
tap349 Selecting previously unselected package chef.
(Reading database ... 146472 files and directories currently installed.)
tap349 Preparing to unpack .../chef_12.6.0-1_amd64.deb ...
tap349 Unpacking chef (12.6.0-1) ...
tap349 Setting up chef (12.6.0-1) ...
tap349 Thank you for installing Chef!
tap349 Starting the first Chef Client run...
tap349 Starting Chef Client, version 12.6.0
tap349 Creating a new client identity for tap349 using the validator key.
tap349 resolving cookbooks for run list: []
tap349 Synchronizing Cookbooks:
tap349 Compiling Cookbooks...
tap349 [2016-01-26T06:31:23-05:00] WARN: Node tap349 has an empty run list.
tap349 Converging 0 resources
tap349
tap349 Running handlers:
tap349 Running handlers complete
tap349 Chef Client finished, 0/0 resources updated in 03 seconds
after bootstrapping new node file tap349.json is created with automatic attributes collected by ohai.
create wrapper cookbooks for community cookbooks:
e.g. for locale
community cookbook
$ cd umka
$ knife cookbook create umka-locale -o cookbooks/
or
$ cd umka/cookbooks
$ berks cookbook umka-locale
or
$ cd umka
$ chef generate cookbook cookbooks/umka-locale
umka/cookbooks/umka-locale/attributes/default.rb:
override['locale']['lang'] = 'en_US.UTF-8'
umka/cookbooks/umka-locale/recipes/default.rb:
include_recipe 'locale'
umka/cookbooks/umka-locale/metadata.rb:
depends 'locale'
umka/Berksfile:
source 'https://supermarket.chef.io'
cookbook 'umka-locale', path: 'cookbooks/umka-locale'
of course it’s possible to store cookbook anywhere - e.g. in default
Chef location ~/.chef/cookbooks/ (knife cookbook create
installs there
by default) or even on github (use git:
option instead of path:
).
in my case wrapper cookbooks are stored in Chef repo for the sake of
convience (though it’s not recommended by Berkshelf - see
https://github.com/berkshelf/berkshelf/issues/535).
$ berks install
Resolving cookbook dependencies...
Fetching 'umka-locale' from source at cookbooks/umka-locale
Fetching cookbook index from https://supermarket.chef.io...
Using build-essential (2.2.4)
Using dmg (2.3.0)
Using chef_handler (1.2.0)
Using git (4.3.6)
Using umka-locale (0.1.0) from source at cookbooks/umka-locale
Installing locale (1.0.2)
Using windows (1.39.1)
Using yum (3.9.0)
Using yum-epel (0.6.5)
Berkshelf automatically installs community cookbooks listed as dependencies of cookbooks inside Berksfile (including wrapper cookbooks) into ~/.berkshelf/cookbooks/.
vendor dependency cookbooks into umka/berks-cookbooks/ so that they are
available for knife-zero
during converge:
$ cd umka
$ berks vendor
Resolving cookbook dependencies...
Fetching 'umka-locale' from source at cookbooks/umka-locale
Fetching cookbook index from https://supermarket.chef.io...
Using umka-locale (0.1.0) from source at cookbooks/umka-locale
Using locale (1.0.2)
Vendoring locale (1.0.2) to /Users/tap/dev/umka/berks-cookbooks/locale
Vendoring umka-locale (0.1.0) to /Users/tap/dev/umka/berks-cookbooks/umka-locale
http://stackoverflow.com/questions/20269623:
Berkshelf in general is very cookbook-centric. Chef, however, is cookbook-repo centric.
add umka/berks-cookbooks/ to cookbook_path
in knife.rb:
cookbook_path ['berks-cookbooks']
you cannot just include ~/.berkshelf/cookbooks/ because it stores installed cookbooks along with their versions - these versions are stripped when vendoring.
add umka/berks-cookbooks/ to .gitignore:
$ cd umka
$ echo 'berks-cookbooks/' >> .gitignore
add new wrapper cookbook to node run_list
:
$ knife node run_list add tap349 umka-locale -z
tap349:
run_list: recipe[umka-locale]
converge:
$ knife zero converge 'name:tap349'
tap349 Starting Chef Client, version 12.6.0
tap349 resolving cookbooks for run list: ["umka-locale"]
tap349 Synchronizing Cookbooks:
tap349 - locale (1.0.2)
tap349 - umka-locale (0.1.0)
tap349 Compiling Cookbooks...
tap349 Converging 3 resources
tap349 Recipe: locale::default
tap349 * apt_package[locales] action install (up to date)
tap349 * execute[Generate locale] action run
tap349 - execute locale-gen en_US.UTF-8
tap349 * execute[Update locale] action run
tap349 - execute update-locale LANG=en_US.UTF-8 LC_ALL=en_US.utf8
tap349
tap349 Running handlers:
tap349 Running handlers complete
tap349 Chef Client finished, 2/3 resources updated in 04 seconds
NOTE: if you make any modification to wrapper cookbooks it’s necessary to vendor and converge again (in loop).
after converging node file is automatically populated with the following:
...
"override": {
"locale": {
"lang": "en_US.UTF-8"
}
},
...
"default": {
"locale": {
"lang": "en_US.utf8",
"lc_all": "en_US.utf8"
}
}
...
it’s not allowed to edit attribute values here as they may be overwritten after chef-client run.
create base role
use roles as kind of lightweight proxy to application cookbooks -
role shouldn’t specify any attributes or have long and detailed run_list
.
umka/roles/base.json:
{
"name": "base",
"description": "Basic server setup",
"chef_type": "role",
"json_class": "Chef::Role",
"default_attributes": {
},
"override_attributes": {
},
"run_list": [
"recipe[umka-base]"
]
}
$ knife cookbook create umka-base -o cookbooks/
umka/cookbooks/umka-base/recipes/default.rb:
include_recipe 'umka-base::system'
include_recipe 'umka-base::packages'
umka/cookbooks/umka-base/recipes/system.rb:
include_recipe 'umka-locale'
include_recipe 'umka-timezone-ii'
umka/cookbooks/umka-base/recipes/packages.rb:
# sudo apt-get update
include_recipe 'apt'
package 'imagemagick'
umka/cookbooks/umka-base/metadata.rb:
...
recipe 'redis::default', 'Installs all non-default recipes'
recipe 'redis::system', 'Installs system packages and configures system'
recipe 'redis::packages', 'Installs common userland packages'
%w(
apt
umka-locale
umka-timezone-ii
).each { |cookbook| depends cookbook }
now umka/nodes/tap349.json can look like this:
"run_list": [
"role[base]"
]
and umka/Berksfile:
source 'https://supermarket.chef.io'
cookbook 'umka-base', path: 'cookbooks/umka-base'
cookbook 'umka-locale', path: 'cookbooks/umka-locale'
cookbook 'umka-timezone-ii', path: 'cookbooks/umka-timezone-ii'
NOTE: it’s still necessary to specify all cookbooks even though the last
2 cookbooks are pulled as dependencies - otherwise berks install
won’t be able to resolve cookbook dependencies.
we need to specify path
because application and wrapper cookbooks are
stored locally (not retrieved from default source - Chef supermarket).
we need to specify all application and wrapper cookbooks in Berksfile
because these cookbooks are later vendored into berks-cookbooks/ directory
- the only cookbooks directory used by
knife
(see .chef/knife.rb) when provisioning the node.
dependencies from cookbook’s metadata.rb are installed (and vendored into berks-cookbooks/) by Berkshelf automatically - but if we need a cookbook from github we need to specify it inside cookbook’s Berksfile and the latter is not processed by Berkshelf when run in the root of chef-repo. thus we need to use a special hack in chef-repo’s Berksfile to load each cookbook’s Berkfile:
source 'https://supermarket.chef.io'
# https://coderwall.com/p/j72egw
def cookbook_dependencies path
berksfile = "#{path}/Berksfile"
return unless File.exists? berksfile
instance_eval File.read(berksfile)
end
Dir['cookbooks/**'].each do |path|
cookbook_dependencies path
cookbook File.basename(path), path: path
end
pay attention to refactored code to add application and wrapper cookbooks.
add environment
e.g. for staging environment:
umka/environments/staging.json:
{
"name": "staging",
"description": "staging environment",
"chef_type": "environment",
"json_class": "Chef::Environment",
"cookbook_versions": {
},
"default_attributes": {
},
"override_attributes": {
"locale": {
"lang": "ru_RU.UTF-8"
}
}
}
NOTE: for some reason chef-client doesn’t find environment if it’s written
in Ruby DSL and you converge with knife-zero
- stick with JSON format.
set environment for node:
$ knife node environment_set tap349 staging -z
get current environment in recipe:
node.chef_environment
NOTE: this is a method - not a property.