Transferring 16tb data between AWS S3 buckets using sync command

DevOps AWS
1

Hi, I’m planning to copy 16tb from one bucket to another. If I use the aws s3 sync command does the data goes via my laptop or the traffic goes directly between buckets?

2

It believe it goes directly between the buckets, wrapping something like the CopyObject API

3

The sync command uses the CopyObject APIs to copy objects between S3 buckets.
https://aws.amazon.com/premiumsupport/knowledge-center/move-objects-s3-bucket/

4

FWIW, when I’m doing large transfers like this, I spin up an EC2 spot instance and run the transfer there

5

Thanks, so there is no option to run the copy in the background

6

For that I’d look at something like S3 Batch Operations: https://aws.amazon.com/s3/features/batch-operations/

7

Thanks, I’ll read that

8

see also bucket replication, if it fits your use case (you can turn on replication for existing objects by contacting support)

9

Another option: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/UsingEMR_s3distcp.html|https://docs.aws.amazon.com/emr/latest/ReleaseGuide/UsingEMR_s3distcp.html

10

you can also copy via aws web console…. but if you are assuming a role, your session will expire