Transferring 16tb data between AWS S3 buckets using sync command

DevOps AWS
#1

Hi, I’m planning to copy 16tb from one bucket to another. If I use the aws s3 sync command does the data goes via my laptop or the traffic goes directly between buckets?

#2

It believe it goes directly between the buckets, wrapping something like the CopyObject API

#3

The sync command uses the CopyObject APIs to copy objects between S3 buckets.
https://aws.amazon.com/premiumsupport/knowledge-center/move-objects-s3-bucket/

#4

FWIW, when I’m doing large transfers like this, I spin up an EC2 spot instance and run the transfer there

#5

Thanks, so there is no option to run the copy in the background

#6

For that I’d look at something like S3 Batch Operations: https://aws.amazon.com/s3/features/batch-operations/

#7

Thanks, I’ll read that

#8

see also bucket replication, if it fits your use case (you can turn on replication for existing objects by contacting support)

#9

Another option: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/UsingEMR_s3distcp.html|https://docs.aws.amazon.com/emr/latest/ReleaseGuide/UsingEMR_s3distcp.html

#10

you can also copy via aws web console…. but if you are assuming a role, your session will expire