Elixir - Testing


test case
test module

$ mix test

notes

mocks

  1. http://blog.plataformatec.com.br/2015/10/mocks-and-explicit-contracts/

Passing the dependency as argument is much simpler and should be preferred over relying on configuration files and Application.get_env/3. When not possible, the configuration system is a good fallback.

Another way to think about mocks is to treat them as nouns. You shouldn’t mock an API (verb), instead you create a mock (noun) that implements a given API.

When you use mock as a verb, you are changing something that already exists, and often those changes are global. When you use mock as a noun, you need to create something new, and by definition it cannot be the SomeDependency module because it already exists.

external APIs

https://github.com/plataformatec/mox/issues/9#issuecomment-423607089

The API you are going to mock is not HTTPoison.get! because that’s not your domain. You are going to mock something like MyApp.TwitterClient.get_tweets which internally calls HTTPoison.get! (or whatever else).

so it’s necessary to mock application boundaries in tests (if they reach these boundaries of course), API clients are your application boundaries => mock API clients (not HTTP client) - say, Lain.API.Google.Drive rather than low-level Lain.API.HTTPClient which is used under the hood of the former.

at the same time it might be necessary to test API clients themselves - there are several ways to do it:

setup vs. setup_all

context variable in setup_all doesn’t have test key:

%{case: MyApp.FooTest, module: MyApp.FooTest}

this makes sense since setup_all is run before the whole suite - not before each test.

about async: true

https://github.com/elixir-lang/elixir/issues/3580#issuecomment-130860923

Tests inside a test case are always run serially but whole cases can run in parallel with other cases with async: true.

=> test cases without async: true are not run in parallel with test cases with async: true.

about async: true in database tests

  1. https://hexdocs.pm/phoenix/testing_schemas.html#test-driving-a-changeset

http://whatdidilearn.info/2018/04/01/testing-phoenix-models-and-controllers.html:

By default, Phoenix uses the Ecto.Adapters.SQL.Sandbox module. Which basically wraps every test within a transaction in order to rollback it after a test is finished. That helps to keep the test database clean.

“Testing Schemas” guide does not recommend us to use async: true option if we are going to interact with the database:

Note: We should not tag any schema case that interacts with a database as :async. This may cause erratic test results and possibly even deadlocks.

Although, Ecto.Adapters.SQL.Sandbox also contains a note about that:

While both PostgreSQL and MySQL support SQL Sandbox, only PostgreSQL supports concurrent tests while running the SQL Sandbox. Therefore, do not run concurrent tests with MySQL as you may run into deadlocks due to its transaction implementation.

We are using PostgreSQL for that project. So I am going to enable that option at my own peril.

[Mox] Mox.expect/4 vs. Mox.stub/3

  1. https://martinfowler.com/articles/mocksArentStubs.html
  2. https://blog.carbonfive.com/2018/01/16/functional-mocks-with-mox-in-elixir

https://stackoverflow.com/a/3459407/3632318

A Mock is just testing behaviour, making sure certain methods are called. A Stub is a testable version (per se) of a particular object.

Mox.expect/4 sets expectations and allows to verify them (that some function in a mock is called exactly n times).

Mox.stub/3 doesn’t set expectations - stubs are never verified (that is it’s possible that stub will never be invoked at all).

as a rule of thumb I use:

https://hexdocs.pm/mox/Mox.html#stub/3

stub is invoked only after all expectations are fulfilled

=> expectations have precedence over stubs: the latter will be used as a fallback when no expectations are set.

[Mox] Mox.stub_with/2

it’s possible to stub all functions in a mock at once by defining a separate stub module and passing it to Mox.stub_with/2:

# test/support/stubs/lain/api/api_stub.ex

defmodule Lain.APIStub do
  @behaviour Lain.API.Behaviour

  @impl true
  def list_page_conversations(_cursor) do
    # return test data
  end
end
# lib/lain/chat/conversation/operations/import.ex

setup do
  Mox.stub_with(Lain.APIMock, Lain.APIStub)
  :ok
end

https://github.com/plataformatec/mox/issues/41#issuecomment-412998957

It (stub_with/2) needs to be inside setup, since setup_all runs in a separate process.

=> you can’t place this line in test/test_helper.exs and rest assured that all calls to mock are now stubbed by default. the problem is that all tests are run in separate processes (regardless of async option value) while mock is stubbed in current process only - in this case in the process in which Mix loads test/test_helper.exs before executing tests (for some unknown reason this process always has #PID<0.91.0> on my machine).

testing GenServers

  1. https://elixir-lang.org/getting-started/mix-otp/genserver.html#testing-a-genserver

start GenServer (Supervisor, etc.):

style guide

  1. https://groups.google.com/forum/#!topic/elixir-ecto/BKpLf092dWs

https://groups.google.com/d/msg/elixir-ecto/BKpLf092dWs/VaCvfZpEBQAJ:

  • test/models/user_test.exs - you will test your changeset and others. Again, only data transformation, tests can be “async: true” because it doesn’t talk to the database.

  • test/models/user_repo_test.exs - you will test anything the model returns that needs the repository to be tested, like complex queries. Here it makes no sense to mock because you need the repo, testing a complex query against a fake repo makes no purpose. Also, since it depends on the Repo, tests cannot be concurrent (until Ecto 1.1 where we solve this problem)

  • test/views/user_test.exs - you will test how your views are rendered. Again, it is only data transformation, so tests can be “async: true” because it doesn’t talk to the database.

  • test/controllers/user_controller_test.exs - integration layer. You’ll test a simple pass through the different layers and the expected result. Important: you are not going to test “models” and “views” combinations here. For example, you won’t test that when you POST attributes A and B, field C will be added to the model. That’s is going to be in the model test. In the same way you are not going to test how users with different attributes are rendered. That’s in the view test.

test names

comparison with RSpec:

test message variants (human messagetest message):

assertions

place result of actual call (result of calling the function under test) on the LHS when comparing it with expected expression:

assert actual == expected

except for the cases when it’s necessary to use pattern matching instead of strict comparison:

assert %{name: "foo"} = MyModule.call()

default values

full vs. partial API result

as a rule JSON API result is either an object (hash, map) or an array of such objects.

tips

add test/support/ to compilation paths

  1. https://github.com/thoughtbot/ex_machina#install-in-just-the-test-environment-for-non-phoenix-projects
  2. https://elixirforum.com/t/load-module-during-test/7400/2

mix.exs:

def project do
  [
    app: :my_app,
    # ...
    elixirc_paths: elixirc_paths(Mix.env)
  ]
end

defp elixirc_paths(:test), do: ["lib", "test/support"]
defp elixirc_paths(_), do: ["lib"]

UPDATE

when generating new Phoenix projects, test/support/ is added to compilation paths by default.

(how to) run specific tests (same as focus in RSpec)

(how to) show detailed information (same as --format documentation in RSpec)

$ mix test --trace

--trace option also sets timeout to infinity - useful when using IEx.pry.

(how to) run tests synchronously

async: true has no effect - useful in case of race conditions.

$ mix test --trace

(how to) use IEx.pry in tests

  1. Elixir - Debugging
  2. https://stackoverflow.com/a/34863997/3632318

to make pry work in tests run mix test inside iex session:

$ alias iex='iex -S mix'
$ iex test --trace

test will time out after 60000 ms by default unless --trace option is used:

** (ExUnit.TimeoutError) test timed out after 60000ms. You can change the timeout:

  1. per test by setting "@tag timeout: x"
  2. per case by setting "@moduletag timeout: x"
  3. globally via "ExUnit.start(timeout: x)" configuration
  4. or set it to infinity per run by calling "mix test --trace"
     (useful when using IEx.pry)

Timeouts are given as integers in milliseconds.

(how to) use Logger in tests

  1. https://stackoverflow.com/a/36349341/3632318

config/test.exs:

- config :logger, level: :warn
+ config :logger, level: :info

this will print all info messages in tests:

Logger.info("foo")

(how to) pass params in controller tests

  1. https://medium.com/@lasseebert/test-driving-a-phoenix-endpoint-part-i-b53e300c1a0a

params can be specified as either keyword list or map:

conn = get(conn, webhook_path(conn, :show), %{"hub.mode" => "subscribe"})
# or
conn = get(conn, webhook_path(conn, :show), ["hub.mode": "subscribe"])

(how to) mock application configuration in tests

  1. https://elixirforum.com/t/using-application-get-env-application-put-env-in-exunit-tests/8019

sometimes it might be necessary to mock application configuration - say, to set some configuration paramater to invalid value.

general recommendations:

client module:

defmodule MyApp.API.AccessToken do
  # ...

  defp refresh_token do
    Application.get_env(:my_app, :api)[:refresh_token]
  end
end

test module:

defmodule MyApp.API.AccessTokenTest do
  use ExUnit.Case, async: false
  # ...

  test "returns error when invalid request (invalid refresh token)" do
    api_config = Application.get_env(:my_app, :api)
    on_exit(fn -> Application.put_env(:my_app, :api, api_config) end)

    new_api_config =
      :my_app
      |> Application.get_env(:api)
      |> put_in([:refresh_token], "foo")

    Application.put_env(:my_app, :api, new_api_config)

    {:error, message} = MyApp.API.AccessToken.get()
    assert "Error refreshing access token: 400: " <> _ = message
  end
end

(how to) mock current environment in tests

  1. Elixir - Tips

see the tip from Elixir - Tips on how to get current environment at runtime.

if you need to test both production and non-production code branches, read read :env configuration paramater dynamically (not via module attribute):

  defmodule MyApp.Foo do
-   @env Application.get_env(:my_app, :env)
-
    def call do
-     if @env == :prod do
+     if Application.get_env(:my_app, :env) == :prod do
        # production code
      else
        # non-production code
      end
    end
  end

then mock :env configuration parameter as described in the tip above.

(how to) test without starting your application and certain deps

  1. https://virviil.github.io/2016/10/26/elixir-testing-without-starting-supervision-tree
  2. https://elixirforum.com/t/ecto-starting-in-test-environment/1205/6
  3. https://hexdocs.pm/ecto/Ecto.Repo.html#c:start_link/1
  4. https://hexdocs.pm/phoenix/Phoenix.Endpoint.html#c:start_link/0

it’s meant to make mix test run faster thanks to not starting certain deps which take a lot of time to start (say, quantum).

don’t start your application and deps (which are applications too) when running tests:

$ mix help test
...
--no-start - does not start applications after compilation
# mix.exs
  defp aliases do
    [
      # ...
-     test: ["ecto.create --quiet", "ecto.migrate", "test"],
+     test: ["ecto.create --quiet", "ecto.migrate", "test --no-start"],
      # ...
    ]
  end

start certain deps along with Repo and Endpoint supervisors manually:

# test/test_helper.exs

# -------------------------------------------------------------------
# start seleted applications only
# -------------------------------------------------------------------

# load application first to get access to its spec:
#
# > https://hexdocs.pm/elixir/Application.html#spec/1
# >
# > Returns nil if the application is not loaded.
Application.load(:lain)

not_started_apps = ~w(
  distillery
  observer_cli
  phoenix_live_reload
  quantum
)a

for app <- Application.spec(:lain, :applications),
    app not in not_started_apps do
  Application.ensure_all_started(app)
end

# -------------------------------------------------------------------
# start required supervisors from Lain.Application manually
# -------------------------------------------------------------------

# for Phoenix application only
#
# > https://elixirforum.com/t/ecto-starting-in-test-environment/1205/6
# >
# > Calling Repo.start_link in test_helper.exs is the correct approach.
Lain.Repo.start_link()
LainWeb.Endpoint.start_link()
# https://hexdocs.pm/elixir/Task.Supervisor.html#start_link/1
Task.Supervisor.start_link(name: Lain.TaskSupervisor)

# -------------------------------------------------------------------
# standard configuration
# -------------------------------------------------------------------

ExUnit.start()

Ecto.Adapters.SQL.Sandbox.mode(Lain.Repo, :manual)

(how to) print query SQL in logs

  1. https://stackoverflow.com/questions/42236123

set log level to debug in test environment:

  # config/test.exs

- config :logger, level: :warn
+ config :logger, level: :debug

(how to) test GenServers and modules using them

if internal message (say, the first periodic message) is used to initialize GenServer state it’s possible to set up everything in test/test_helper.exs:

# test/test_helper.exs

# -------------------------------------------------------------------
# - start required workers from Lain.Application manually
# - allow them to use stubs defined in current test process
# - wait till they are ready
# -------------------------------------------------------------------

{:ok, child_pid} = Lain.FB.Label.Sync.start_link([])

Lain.APIMock
|> Mox.stub_with(Lain.APIStub)
|> Mox.allow(self(), child_pid)

# initialize state manually by sending the first periodic message
send(child_pid, :work)
:synced = :sys.get_state(child_pid)

# or else by making a synchronous call - if specified message is
# handled by handle_call/3 callback of course
:ok = GenServer.call(Lain.FB.Label.Sync, :sync_now)

(how to) suppress logs in tests

  1. https://minhajuddin.com/2019/02/01/pearls-of-elixir-interesting-patterns-from-popular-elixir-packages/
  defmodule Francesca.FB.Ad.Operations.CollectDailyAdStatsTest do
    use Francesca.DataCase
    import Mox

+   @moduletag :capture_log

    # ...
  end

troubleshooting

(DBConnection.OwnershipError) cannot find ownership process

$ mix test
...
== Compilation error in file test/gertruda/ga/ecomm_stat/query_test.exs ==
** (DBConnection.OwnershipError) cannot find ownership process for #PID<0.473.0>.

solution

you must have forgotten to use test macro (and no test process was started).