Ecto
27 Jul 2017- migrations
- repositories
- schemas
- changesets
- associations
- models vs. changesets
- Ecto.Multi
- [Programming Phoenix] prefixes
- embeds
- pipe vs. keyword syntax
- JSONB
- upsert
- https://hexdocs.pm/phoenix/ecto.html
- https://hexdocs.pm/ecto/getting-started.html
- https://hexdocs.pm/ecto/Ecto.html
migrations
create migration:
$ mix ecto.gen.migration add_payment_service_to_transfers
run all pending migrations:
$ mix ecto.migrate
rollback last applied migration:
$ mix ecto.rollback
NOTE: there is nothing like schema.rb file in Phoenix project - database schema is not dumped to file after running migrations.
primitive column types
- https://hexdocs.pm/ecto/Ecto.Schema.html#module-types-and-casting
- https://hexdocs.pm/phoenix/Mix.Tasks.Phx.Gen.Schema.html#module-attributes
repositories
https://hexdocs.pm/phoenix/ecto.html:
Our repo (MyApp.Repo) has three main tasks:
- to bring in all the common query functions from Ecto.Repo
- to set the otp_app name equal to our application name
- to initialize the options passed to the database adapter in init/2
schemas
schemas implement Queryable
protocol - that is why it’s possible to pass
them to Ecto.Repo
functions.
changesets
- https://hexdocs.pm/phoenix/ecto.html
- http://cultofmetatron.io/2017/04/22/thinking-in-ecto---schemas-and-changesets/
def changeset(%User{} = user, attrs) do
user
|> cast(attrs, [:uuid]) # params.require(:user).permit(:uuid) (strong parameters)
|> validate_required([:uuid]) # validates :uuid, presence: true (AM validations)
end
cast/3
permits and casts parameters to types defined in schema.
changeset vs. Ecto.Changeset.change/2
Ecto.Changeset.change/2
wraps schema inside a changeset (and optionally
adding changes) but it DOESN’T call changeset
function of this schema =>
no validations are run and resulting changeset is always valid.
changeset vs. schema struct
“Improved associations and factories” chapter, “Less changesets” section of “What’s new in Ecto 2.1” book
You can now also pass structs to the repository and Ecto will take care of building the changesets for you behind the scenes.
https://github.com/elixir-ecto/ecto/issues/2197#issuecomment-326336374
The struct is converted to a Changeset using Ecto.Changeset.change/2, not your own function.
it’s possible to pass both changeset and schema, say, to Repo.insert/2
.
in case of changeset, this changeset can be either custom schema changeset
or default changeset created by Ecto.Changeset.change/2
.
in case of struct, this struct is still converted to changeset but always
with Ecto.Changeset.change/2
- all validations defined in custom schema
changeset become unavailable and error will be raised only if underlying
data store returns error.
associations
associations vs. FK columns
prefer defining associations to specifying FK columns in schema definitions (or else you won’t be able to use them in queries):
schema "cards" do
# define association:
belongs_to :user, MyApp.User
# instead of specifying FK column:
#field :user_id, :id
timestamps()
end
add existing association in changeset function
-
put association using
Ecto.Changeset.put_assoc/4
@create_permitted_fields ~w(body)a @required_fields ~w(body)a def changeset(%__MODULE__{} = comment, attrs) do comment |> cast(attrs, @create_permitted_fields) |> validate_required(@required_fields) |> put_assoc(:post, attrs[:post]) end
-
set association ID directly
@create_permitted_fields ~w(body post_id)a @required_fields ~w(body post_id)a def changeset(%__MODULE__{} = comment, attrs) do comment |> cast(attrs, @create_permitted_fields) |> validate_required(@required_fields) |> foreign_key_constraint(:post_id) end
Ecto.Changeset.foreign_key_constraint/3
is used to add Ecto error to changeset if foreign key constraint violation occurred in the database (it is akin toEcto.Changeset.unique_constraint/3
which also checks for unique constraint violation by relying on the database).
preloading associations
assocations are never preloaded by default - load them explicitly!
associations can be preloaded in:
-
in query when loading parent struct
Repo.all(from p in Post, preload: [:comments])
-
a posteriori after loading parent struct
posts = Repo.all(Post) |> Repo.preload(:comments)
associations can be loaded in:
-
into parent struct
post = Repo.preload(post, :comments)
-
into its own struct
query = Ecto.assoc(post, :comments) comments = Repo.all(query)
building associations
put_assoc
vs. cast_assoc
- https://medium.com/coryodaniel/til-elixir-ecto-put-assoc-vs-cast-assoc-7c80f35f6e6
- https://hexdocs.pm/ecto/Ecto.Changeset.html#cast_assoc/3
-
when using
put_assoc
:association that you
put
usually already exists (that is it’s not meant to be persisted as a result of current operation) and is supplied as a struct or changeset (say,user
when creating acomment
). -
when using
cast_assoc
:you specify the name of association that is expected to be created alongside its parent - this name is also used to fetch association params from supplied
params
) - like when usingaccepts_nested_attributes_for
in Rails.
however in both cases corresponding association must be preloaded so that Ecto
knows what to do in case it already exists (of course it only makes sense when
parent has id
field - but it doesn’t raise error otherwise).
build_assoc
vs. building association explicitly
# build association using parent struct and association name
user = Repo.get_by(User, name: "John")
comment = Ecto.build_assoc(user, :comments, body: "foo")
=
# build association explicitly by setting FK column value
user = Repo.get_by(User, name: "John")
comment = %Comment{user_id: user.id, body: "foo"}
The goal of build_assoc is to allow you to work on the association names instead of the key names.
usage examples of put_assoc
, build_assoc
and cast_assoc
it’s not recommended to use these functions inside schemas since they might
require you to make calls to Repo
(build_assoc
or put_assoc
) or use
changesets of other schemas (cast_assoc
) while it’s preferable to isolate
both calls to Repo
and usage of different changesets in contexts.
though in What's new in Ecto 2.1
ebook (chapter Many to many and upserts
)
there is an example of schema in which both put_assoc
and Repo
are used.
-
put_assoc
https://elixirforum.com/t/using-put-assoc-in-changeset-for-many-to-many/3848/2:
However, that put_assoc is about a belongs_to relationship and that expects either nil or a struct.
build new comment using its
user
association and existing user:# in controller or context user = Repo.get_by(User, name: "John") comment |> Repo.preload(:user) |> Comment.changeset(params) |> put_assoc(:user, user) |> Repo.insert!() # in schema def changeset(%Comment{} = comment, params) do comment |> cast(params, [:body]) |> validate_required([:body]) end
-
build_assoc
build new comment using existing user and his
comments
association:# in controller or context user = Repo.get_by(User, name: "John") user |> build_assoc(:comments) |> Comment.changeset(params) |> Repo.insert!() # in schema def changeset(%Comment{} = comment, params) do comment |> cast(params, [:body]) |> validate_required([:body]) end
-
cast_assoc
build new user with new comments to save them all later in one go:
# in controller or context user = %User{} user |> Repo.preload(:comments) |> User.changeset(params) |> cast_assoc(:comments, with: &Comment.insert_changeset/2) |> Repo.insert!() # in schema # &Comment.changeset/2 is used by default # to create changesets for comments def changeset(%User{} = user, params) do user |> cast(params, [:name]) |> validate_required([:name]) end
models vs. changesets
http://blog.tokafish.com/rails-to-phoenix-getting-started-with-ecto/:
Typically, you’d work with a changeset for making modifications to a model via the repo, and you’d work with the model when fetching the data for display.
UPDATE
models have been deprecated in Phoenix 1.3 - they are called schemas now.
Ecto.Multi
- https://hexdocs.pm/ecto/Ecto.Multi.html
- http://blog.danielberkompas.com/2016/09/27/ecto-multi-services.html
- https://medium.com/@feymartynov/an-example-of-using-ecto-multi-5f7fc8cf3cc1
- replace
before
callbacks with functions in changesets - replace
after
callbacks with operations inEcto.Multi
use it when you would need callbacks in AR: Ecto.Multi
allows to pack functions
that should be called after main action (like create
or update
) - all these
functions are named operations in Ecto.Multi
parlance.
moreover using Ecto.Multi
allows to stop execution if some operation fails and
returns {:error, reason}
- this error can be returned either automatically
from Repo
function or manually from functions passed to Ecto.Multi.run
.
in the end Ecto.Multi
struct is usually passed to Repo.transaction/1
which
rollbacks transaction if any operation fails however calling Repo.transaction/1
in the end is not obligatory (if you don’t want all operations to be run in
transaction - for example, they write to filesystem which cannot be rolled back).
it all resembles using monads to handle errors in general and using
dry-monads
and dry-matcher
gems in Ruby in particular
(monads in Ruby).
but unlike dry-matcher
Ecto.Multi
stores additional information in case of
failure - not only do we have error itself but also operation name and changes
accumulated in previous succeeded operations.
in this regard Ecto.Multi
acts more like dry-transaction
gem which allows to
handle errors arising from particular steps (= operations) with the difference
that dry-transaction
holds operations (service objects that respond to call
)
from DI container while multi packs operations (Repo
or arbitrary functions).
also don’t forget that there exist several monad libraries for Elixir
(e.g. MonadEx) which allow to use
this error handling mechanism in any module while Ecto.Multi
is used when
you deal with persistence and need something to replace callbacks
(that is Ecto.Multi
is alternative to our custom operations in Rails projects
which both persist data and run after_*
callbacks manually).
https://github.com/elixir-ecto/ecto/issues/1114#issuecomment-162985202:
if it’s necessary to get access to result of previous operation it’s necessary
to use run
functions - say, if you first insert a record and then need to
updated it you cannot use Ecto.Multi.update
in the 2nd case since you won’t
be able to get previously inserted record.
https://elixirforum.com/t/ecto-multi-without-transaction/7453:
it’s not possible to execute Ecto.Multi
operations outside of transaction -
use with
statement instead.
[Programming Phoenix] prefixes
NOTE: it’s not possible to perform joins across prefixes - data in different prefixes must be completely isolated.
prefixes can be configured on different levels:
global prefix -> schema prefix -> query/repository operation/struct prefix
global prefix
config/dev.exs:
config :rumbl, Rumbl.Repo,
adapter: Ecto.Adapters.Postgres,
username: "postgres",
password: "postgres",
database: "rumbl_dev",
hostname: "localhost",
pool_size: 10,
after_connect: {Postgrex, :query!, ["SET search_path TO new_prefix", []]}
schema prefix
NOTE: since Ecto 2.1
defmodule Rumbl.User do
# model/0 in web/web.ex uses Ecto.Schema:
# use Ecto.Schema
use Rumbl.Web, :model
@schema_prefix "new_prefix"
schema "users" do
field :name, :string
field :username, :string
field :password, :string, virtual: true
field :password_hash, :string
has_many :videos, Rumbl.Video
timestamps
end
query prefix
> query = Ecto.Queryable.to_query Rumbl.User
#Ecto.Query<from u in Rumbl.User>
> Rumbl.Repo.all %{query | prefix: "new_prefix"}
[]
repository operation prefix
> Rumbl.Repo.all Rumbl.User, prefix: "new_prefix"
struct prefix
if it’s nil it means global prefix is used (“public” or as configured in environment config).
> [video] = Rumbl.Repo.all Rumbl.Video
[%Rumbl.Video{__meta__: #Ecto.Schema.Metadata<:loaded, "videos">, ...}
> new_prefix_video = Ecto.put_meta(video, prefix: "new_prefix")
%Rumbl.Video{__meta__: #Ecto.Schema.Metadata<:loaded, "new_prefix", "videos">, ...}
embeds
- http://blog.plataformatec.com.br/2015/08/working-with-ecto-associations-and-embeds/
- https://robots.thoughtbot.com/embedding-elixir-structs-in-ecto-models
- https://elixirforum.com/t/cast-embed-causing-error/6678/3
updating
you cannot update embed unless you set on_replace: :update
option:
embeds_one :data, TransferData, on_replace: :update
otherwise calling cast_embed(changeset, :data)
will raise error:
** (RuntimeError) you are attempting to change relation :data of
Billing.App.Transfer but the `:on_replace` option of
this relation is set to `:raise`.
after that you are allowed to pass updated fields as a map only:
** (RuntimeError) you have set that the relation :data of Billing.App.Transfer
has `:on_replace` set to `:update` but you are giving it a struct/
changeset to put_assoc/put_change.
Since you have set `:on_replace` to `:update`, you are only allowed
to update the existing entry by giving updated fields as a map or
keyword list or set it to nil.
when using on_replace: :update
you cannot update embed by passing struct
or changeset - use map or keywords list of fields instead:
** (exit) an exception was raised:
** (RuntimeError) you have set that the relation :data of Billing.App.Transfer
has `:on_replace` set to `:update` but you are giving it a struct/
changeset to put_assoc/put_change.
Since you have set `:on_replace` to `:update`, you are only allowed
to update the existing entry by giving updated fields as a map or
keyword list or set it to nil.
- transfer = md |> App.get_own_transfer_by_md!(user)
- # don't use atom key for pares or else it can be accessed
- # using only atom key in API.complete_transfer/1 - key will
- # be stringified only when transfer is read from DB again
- data = Map.put(transfer.data, "pares", attrs["pares"])
-
- transfer
- |> Transfer.update_changeset(%{data: data})
- |> Repo.update()
+ transfer = md |> App.get_own_transfer_by_md!(user)
+ attrs = %{data: %{pares: pares}}
+
+ transfer
+ |> Transfer.update_changeset(attrs)
+ |> Repo.update()
complete example:
iex> transfer = Billing.Repo.get(Billing.App.Transfer, 11)
iex> attrs = %{data: %{acs_url: "foo"}}
iex> transfer |> cast(attrs, []) |> cast_embed(:data) |> Repo.update()
nested embeds
lib/billing/app/transfer.ex:
defmodule Billing.App.Transfer do
use Ecto.Schema
import Ecto.Changeset
alias Billing.App.{Card, TransferData}
schema "transfers" do
field :fee, :decimal
embeds_one :data, TransferData, on_replace: :update
timestamps type: :utc_datetime
end
# > casting embeds with cast/4 is not supported,
# > use cast_embed/3 instead
@doc false
def update_changeset(%__MODULE__{} = transfer, attrs) do
transfer
|> cast(attrs, [:fee])
|> cast_embed(:data)
end
end
lib/billing/app/transfer_data.ex:
defmodule Billing.App.TransferData do
use Ecto.Schema
import Ecto.Changeset
@primary_key false
embedded_schema do
field :acs_url, :string
embeds_one :error, Error, primary_key: false, on_replace: :update do
field :service, :integer
end
end
# changeset/2 is called by cast_embed/3 by default
def changeset(%__MODULE__{} = data, attrs) do
data
|> cast(attrs, [:acs_url])
|> cast_embed(:error, with: &errors_changeset/2)
end
defp errors_changeset(%__MODULE__.Error{} = error, attrs) do
error
|> cast(attrs, [:service])
end
end
pipe vs. keyword syntax
- pipe syntax
- pipe-based syntax, macro-based syntax, expression syntax, pipe expression syntax
- keyword syntax
- keyword-based syntax, keyword query syntax
# https://github.com/elixir-ecto/ecto
defmodule Sample.App do
import Ecto.Query
alias Sample.{Weather, Repo}
def pipe_query do
Weather
|> where(city: "Tula")
|> order_by(:temp_lo)
|> limit(10)
|> Repo.all()
end
def keyword_query do
query = from w in Weather,
where: w.prcp > 0 or is_nil(w.prcp),
select: w
Repo.all(query)
end
end
IMO pipe syntax is more suitable for simple queries:
User |> order_by(desc: :inserted_at)
# reads better than
from u in User, order_by: [desc: u.inserted_at]
while keyword syntax is geared towards complex queries.
JSONB
use map
Ecto type for JSONB fields.
https://elixirforum.com/t/how-do-i-use-the-postgres-jsonb-postgrex-json-extension/3214/7
…my question is about “map” vs “jsonb”
They are equivalent when using postgres. map is an abstract type and each database adapter can choose the actual representation - for postgres it’s jsonb.
default value
default value can be set in either migration or schema or both but in general
it’s recommended to set default value in migration only unless it’s very
application specific (not just false
or 0
) or you need different default
values in different schemas for the same table in DB.
-
migration
it’s necessary to use a string “{}” or “[]” as default value - this default value must be passed as is to PostgreSQL (it treats
%{}
or[]
as NULL).# priv/repo/migrations/20181012131944_create_events.exs create table(:events) do # https://elixirforum.com/t/5899/4 add :data, :map, null: false, default: "{}" end
-
schema
note that it’s necessary to use
%{}
or[]
unlike in migration:# lib/lain/log/event/event.ex schema "events" do field :data, :map, default: %{} end
(how to) check if JSONB column is empty
JSONB column is empty if it contains empty object or empty array:
MyApp.User
|> where([u], u.data == fragment("'{}'::jsonb"))
|> MyApp.Repo.all()
MyApp.User
|> where([u], u.data == fragment("'[]'::jsonb"))
|> MyApp.Repo.all()
upsert
- https://hexdocs.pm/ecto/Ecto.Repo.html#c:insert/2-upserts
- https://github.com/elixir-ecto/ecto/issues/2181#issuecomment-324325516
- https://github.com/elixir-ecto/ecto/issues/2382
Upsert will not change ID primary key of already existing rows, just update other fields.
We should replace all except the primary key.
=> upsert replaces all fields except for ID (including inserted_at)!
UPDATE (2018-11-08)
- http://blog.plataformatec.com.br/2018/10/a-sneak-peek-at-ecto-3-0-performance-migrations-and-more
- https://hexdocs.pm/ecto/Ecto.Repo.html#c:insert/2-upserts
it’s now possible to replace only certain fields in Ecto 3:
on_conflict: {:replace, [:foo, :bar, :baz]}
when using this option:
inserted_at
field is not updated- it’s necessary to list
updated_at
field explicitly for it to be updated
all replace
options:
https://hexdocs.pm/ecto/Ecto.Repo.html#c:insert/2-upserts
:replace_all replace all values on the existing row with the values in the schema/changeset, including autogenerated fields such as inserted_at and updated_at
:replace_all_except_primary_key same as above except primary keys are not replaced
replace only specific columns. This option requires conflict_target
replace_all
tries to replace ALL fields including timestamps, primary and
foreign keys - if you try to insert conflicting association without parent ID
already set (say, via cast_assoc
), you’ll get Ecto.ConstraintError
error:
** (Ecto.ConstraintError) constraint error when attempting to insert struct:
* messages_conversation_id_fkey (foreign_key_constraint)