A measurement of the associated production of a top-quark pair (tt) with a vector boson (W, Z) in proton-proton collisions at a center-of-mass energy of 13 TeV is presented, using 36.1 fb-1 of integrated luminosity collected by the ATLAS detector at the Large Hadron Collider. Events are selected in channels with two same- or opposite-sign leptons (electrons or muons), three leptons or four leptons, and each channel is further divided into multiple regions to maximize the sensitivity of the measurement. The ttZ and ttW production cross sections are simultaneously measured using a combined fit to all regions. The best-fit values of the production cross sections are σttZ=0.95±0.08stat±0.10syst pb and σttW=0.87±0.13stat±0.14syst pb in agreement with the Standard Model predictions. The measurement of the ttZ cross section is used to set constraints on effective field theory operators which modify the ttZ vertex.