CircleCI
31 May 2018- SSH
- caching
- troubleshooting
- build is not starting after changes are pushed to GitHub
- /bin/bash: yarn: command not found
- [Rails] Rails.application.credentials are empty in Rails 5.2+
- deploy job prompts for deploy user password
- [Phoenix] test.secret.exs is missing on CircleCI
- [Phoenix] build cache from build stage is not used in deploy stage
- [Rails] Unable to fetch some archives
- You must use Bundler 2 or greater with this lockfile.
SSH
https://circleci.com/docs/1.0/permissions-and-access-during-deployment/
The ssh private keys that you add from the page are stored under the ~/.ssh directory for the build user in the container. We also add entries to ~/.ssh/config to specify which key is used to access which host. For example, if you add a key with the hostname prod-server, then ~/.ssh/id_prod-server will be automatically created and ~/.ssh/config will have an entry that looks like:
Host prod-server IdentitiesOnly yes IdentityFile /home/ubuntu/.ssh/id_prod-server
SSH config is not available on CI/CD server (unless you copy it explicitly) => use IP address instead of SSH config entry name to specify hostname:
- when adding your private key in project settings on CircleCI
- in deploy config of your application (say, .deliver/config or config/deploy/production.rb)
caching
cache archives are uploaded to external storage and are available across different containers (say, you can save cache in build job and restore it later in deploy job - jobs are using separate Docker containers).
partial cache restoration
CircleCI 2.0 caching finally clicked with me after reading this comment:
https://discuss.circleci.com/t/circle-2-0-caching-is-too-limited-to-be-very-useful/11694/2
- restore_cache: keys: - projectname-npm-deps-{{ .Branch }}-{{ checksum "package.json" }} - projectname-npm-deps-{{ .Branch }} - projectname-npm-deps-
After the restore_cache step above you run npm install and npm will ‘fill in the gaps’. So let’s say we were only able to restore the last (least specific) cache above - it’s likely that most of your dependencies will be similar and npm install will figure out what else needs installing.
cache invalidation
say, we have these keys in restore_cache
and save_cache
steps:
- restore_cache:
keys:
- v1-mix-cache-{{ checksum "mix.lock" }}
- v1-mix-cache-{{ .Branch }}
- v1-mix-cache
# ...
- save_cache:
key: v1-mix-cache-{{ checksum "mix.lock" }}
paths: deps
- save_cache:
key: v1-mix-cache-{{ .Branch }}
paths: deps
- save_cache:
key: v1-mix-cache
paths: deps
existing caches are NEVER updated or invalidated!
say, mix.lock is updated and its checksum is changed.
-
restore_cache
stepCircleCI expands templates in all specified keys and tries to find existing keys (that is keys stored somewhere in its internal KV store). the 2nd and the 3rd cache keys are found (checksum has changed, branch stays the same, the 3rd key always matches unless you increment a version prefix). then the most recently generated cache is restored and used - not the cache under the most specific matching key (though it’s highly likely that the most recently generated cache is stored under the most specific matching key).
-
save_cache
stepCircleCI expands templates in all
save_cache
keys and saves current cache under all NEW cache keys (that is keys which are not present in KV store) - this is the 1st key only since only checksum has changed. if, say, not only is mix.lock updated but branch is switched as well, current cache will be saved under both the 1st and the 2nd cache keys.
troubleshooting
build is not starting after changes are pushed to GitHub
solution
there was an error in my CircleCI config - version: 2
key was missing in
workflows
section.
I have found this error by clicking project name (or else it’s possible to click any branch of the project):
CircleCI |
---|
BUILDS (left menu) → <project_name> (link) |
you’ll see a big yellow NEEDS SETUP
button if there are any config errors.
/bin/bash: yarn: command not found
solution
pre-built CircleCI Docker image circleci/ruby:2.5.1
doesn’t have Yarn
preinstalled - use -node
image variant instead:
https://circleci.com/docs/2.0/yarn/#using-yarn-in-circleci
If you are using one of the other language images such as circleci/python or circleci/ruby, there are two image variants that will include Yarn as well as NodeJS. These would be the -node and -node-browsers image variants.
[Rails] Rails.application.credentials are empty in Rails 5.2+
solution
Rails 5.2 uses encrypted config/credentials.yml.enc to store credentials and config/master.key to decrypt it but the latter is not available on CircleCI.
it’s possible to store master key in RAILS_MASTER_KEY
environment variable:
http://guides.rubyonrails.org/5_1_release_notes.html#encrypted-secrets
Secrets will be decrypted in production, using a key stored either in the RAILS_MASTER_KEY environment variable, or in a key file.
CircleCI |
---|
Project Settings → BUILD SETTINGS (section) → Environment Variables |
deploy job prompts for deploy user password
solution
- https://circleci.com/docs/2.0/configuration-reference/#add_ssh_keys
- https://support.circleci.com/hc/en-us/articles/115015628247-Are-you-sure-you-want-to-continue-connecting-yes-no-
I haven’t added my private SSH key:
CircleCI |
---|
Project Settings → PERMISSIONS (section) → SSH permissions |
Add SSH key (button) |
Hostname
(input): production server IPPrivate Key
(textarea): contents of your ~/.ssh/id_rsa
after SSH key is added, you’ll see its fingerprint - use it to add SSH key in
add_ssh_keys
step of deploy
job in CircleCI config:
# .circleci/config.yml
jobs:
deploy:
steps:
- checkout
+ - add_ssh_keys:
+ fingerprints:
+ - '21:cd:07:a3:b8:b9:1f:20:b0:eb:59:c7:e3:14:fc:5e'
NOTE: it’s possible not to specify which SSH keys to use (via fingerprints) explicitly in CircleCI config - by default all added SSH keys will be loaded:
https://circleci.com/docs/2.0/configuration-reference/#add_ssh_keys
Note that CircleCI 2.0 jobs are auto configured with ssh-agent with all keys auto-loaded, and is sufficient for most cases. add_ssh_keys may be needed to have greater control over which SSH keys to authenticate
[Phoenix] test.secret.exs is missing on CircleCI
by default config/test.secret.exs is added to .gitignore and consequently ignored by Git (untracked). still it’s imported in config/test.exs:
# config/test.exs
import_config "test.secret.exs"
as a result it’s not found on CircleCI:
$ #!/bin/bash -eo pipefail
mix local.hex --if-missing --force
** (Code.LoadError) could not load /home/circleci/app/config/test.secret.exs
solution
-
create a separate config/test.secret.exs.circleci
this file shouldn’t contain any sensitive information but database credentials only:
# config/test.secret.exs.circleci use Mix.Config config :lain, Lain.Repo, username: "lain_test", password: "lain_test", database: "lain_test", hostname: "localhost", pool_size: 10, pool: Ecto.Adapters.SQL.Sandbox
config/test.secret.exs.circleci can be safely added to Git repo and pushed to GitHub.
-
copy it to config/test.secret.exs on CircleCI
config/test.secret.exs must be created right after
checkout
step in bothbuild
anddeploy
jobs:# .circleci/config.yml version: 2 jobs: build: # ... steps: - checkout + # ------------------------------------------------------------- + # Create test.secret.exs + # ------------------------------------------------------------- + + - run: cp config/test.secret.exs{.circleci,} + # ... deploy: # ... steps: - checkout + # ------------------------------------------------------------- + # Create test.secret.exs + # ------------------------------------------------------------- + + - run: cp config/test.secret.exs{.circleci,} + # ...
[Phoenix] build cache from build stage is not used in deploy stage
solution
it has turned out I use different Mix environments in build and deploy stages so the cache from the former is of no use for the latter.
there might be 2 solutions:
- use the same Mix environment (
MIX_ENV
environment variable) in both stages - save cache in both stages (currently I don’t save cache in deploy stage)
[Rails] Unable to fetch some archives
# .circleci/config.yml
jobs:
build:
steps:
- run:
name: Install psql
command: sudo apt install postgresql-client
Install psql
step log:
Get:5 http://security.debian.org stretch/updates/main amd64 postgresql-client all 9.6+181+deb9u1 [55.7 kB]
Fetched 168 kB in 0s (485 kB/s)
E: Failed to fetch http://deb.debian.org/debian/pool/main/p/postgresql-9.6/postgresql-client-9.6_9.6.7-0+deb9u1_amd64.deb 404 Not Found
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
Exited with code 100
solution
# .circleci/config.yml
jobs:
build:
steps:
+ - run:
+ # or else postgresql-client package is not found
+ name: Update package list
+ command: sudo apt-get update
+
- run:
name: Install psql
command: sudo apt install postgresql-client
You must use Bundler 2 or greater with this lockfile.
# .circleci/config.yml
jobs:
build:
steps:
- run:
name: Bundle install
command: bundle install --jobs=4 --retry=3 --path vendor/bundle
Bundle install
step log:
You must use Bundler 2 or greater with this lockfile.
solution
# .circleci/config.yml
jobs:
build:
docker:
- image: circleci/ruby:2.5.0
+ environment:
+ BUNDLER_VERSION: 2.0.1
+ steps:
+ - run:
+ name: Update bundler
+ command: gem update bundler
+
- run:
name: Bundle install
command: bundle install --jobs=4 --retry=3 --path vendor/bundle
deploy:
docker:
- image: circleci/ruby:2.5.0
+ environment:
+ BUNDLER_VERSION: 2.0.1
+ steps:
+ - run:
+ name: Update bundler
+ command: gem update bundler
+
- run:
name: Bundle install
command: bundle install --jobs=4 --retry=3 --path vendor/bundle