aws-in-a-box

This project is an emulator for several AWS services, à la Localstack. README in progress!

Currently supported services (see below for full support details):

• DynamoDB - highly experimental, only enough for Kinesis Client Library to work. Not recommended to use!
• Kinesis
• KMS
• S3
• SQS

Aws-in-a-box runs on HTTP (not HTTPS) but supports HTTP2 upgrade with h2c (HTTP without TLS).

Why use this over localstack?

• High-performance; no overhead from docker or proxies
• Single statically-linked 7MB native binary. No interpereter/runtime hell. (There are also 3MB compressed docker images if you prefer)
• Simple persistence scheme using atomic file writes. When using the native binary, no more broken persistent docker volumes

Usage

The easiest way to consume this project is from pre-built artifacts on the release page or the docker image. You can also build from source with either go native tooling or with Bazel, see Development section.

-addr string
        Address to run on (default "localhost:4569")
  -reuse-port
      If set, will apply the SO_REUSEPORT socket option on Darwin/Linux platforms.
  -enableKMS
        Enable KMS service (default true)
  -enableKinesis
        Enable Kinesis service (default true)
  -enableSQS
        Enable SQS service (default true)
  -enableS3
      Enable S3 service (default true)
  -experimental_enableDynamoDB
        Enable DynamoDB service (default true)
  -kinesisDefaultDuration duration
        How long to retain messages. Can be used to control memory usage. After creation, retention can be adjusted with [Increase/Decrease]StreamRetentionPeriod (default 24h0m0s)
  -kinesisInitialShardsPerStream int
        How many shards to create for each stream listed in -kinesisInitialStreams (default 2)
  -kinesisInitialStreams string
        Streams to create at startup. Example: stream1,stream2,stream3
  -kinesisStreamCreateDuration duration
        How long a new Kinesis stream stays in CREATING status (default 5s)
  -kinesisStreamDeleteDuration duration
        How long a deleted Kinesis stream stays in DELETING status (default 5s)
  -logLevel string
        debug/info/warn/error (default "debug")
  -persistDir string
        Directory to persist data to. If empty, data is not persisted.
  -s3InitialBuckets string
        Buckets to create at startup. Example: bucket1,bucket2,bucket3

Development

You can use either native go tooling or Bazel.

Running the service

• go run .
• bazel run //:aws-in-a-box

Running tests

• go test ./...
• bazel test //...
  

Kinesis Support

Most of Kinesis is implemented, including the Consumer APIS. Remaining work:

• KMS integration not wired up
• A few List/Describe APIs are missing
• Shard split/merge

There is no persistence for Kinesis data.

Click to expand the detailed support table

API	Support Status	Caveats/Notes
AddTagsToStream	✅ Supported
CreateStream	✅ Supported
DecreaseStreamRetentionPeriod	✅ Supported
DeleteStream	✅ Supported
DeregisterStreamConsumer	✅ Supported
DescribeLimits	❌ Unsupported
DescribeStream	❌ Unsupported	This API is discouraged by AWS
DescribeStreamConsumer	✅ Supported
DescribeStreamSummary	✅ Supported
DisableEnhancedMonitoring	❌ Unsupported	Cloudwatch not implemented
EnableEnhancedMonitoring	❌ Unsupported	Cloudwatch not implemented
GetRecords	✅ Supported
GetShardIterator	✅ Supported
IncreaseStreamRetentionPeriod	✅ Supported
ListShards	✅ Supported
ListStreamConsumers	❌ Unsupported
ListStreams	✅ Supported
ListTagsForStream	✅ Supported
MergeShards	❌ Unsupported	No support for merging/splitting yet.
PutRecord	✅ Supported
PutRecords	✅ Supported
RegisterStreamConsumer	✅ Supported
RemoveTagsFromStream	✅ Supported
SplitShard	❌ Unsupported	No support for merging/splitting yet.
StartStreamEncryption	❌ Unsupported
StopStreamEncryption	❌ Unsupported
SubscribeToStream	✅ Supported
UpdateShardCount	❌ Unsupported	No support for merging/splitting yet.
UpdateStreamMode	❌ Unsupported

KMS Support

Most of KMS is implemented. Remaining work:

• Key deletion/key rotation is missing
• Grants are missing
• Key policies are missing

KMS data is fully persisted.

Click to expand the detailed support table

API	Support Status	Caveats/Notes
CancelKeyDeletion	❌ Unsupported
ConnectCustomKeyStore	❌ Unsupported
CreateAlias	✅ Supported
CreateCustomKeyStore	❌ Unsupported
CreateGrant	❌ Unsupported
CreateKey	✅ Supported	ECC_SECG_P256K1 and SM2 not supported
Decrypt	✅ Supported
DeleteAlias	✅ Supported
DeleteCustomKeyStore	❌ Unsupported
DeleteImportedKeyMaterial	❌ Unsupported
DescribeCustomKeyStores	❌ Unsupported
DescribeKey	✅ Supported	Lots of metadata properties missing
DisableKey	✅ Supported
DisableKeyRotation	❌ Unsupported
DisconnectCustomKeyStore	❌ Unsupported
EnableKey	✅ Supported
EnableKeyRotation	❌ Unsupported
Encrypt	✅ Supported
GenerateDataKey	✅ Supported
GenerateDataKeyPair	✅ Supported	ECC_SECG_P256K1 not supported
GenerateDataKeyPairWithoutPlaintext	✅ Supported	ECC_SECG_P256K1 not supported
GenerateDataKeyWithoutPlaintext	✅ Supported
GenerateMac	✅ Supported
GenerateRandom	✅ Supported
GetKeyPolicy	❌ Unsupported
GetKeyRotationStatus	❌ Unsupported
GetParametersForImport	❌ Unsupported
GetPublicKey	❌ Unsupported
ImportKeyMaterial	❌ Unsupported
ListAliases	✅ Supported
ListGrants	❌ Unsupported
ListKeyPolicies	❌ Unsupported
ListKeys	✅ Supported
ListResourceTags	✅ Supported
ListRetirableGrants	❌ Unsupported
PutKeyPolicy	❌ Unsupported
ReEncrypt	✅ Supported
ReplicateKey	❌ Unsupported
RetireGrant	❌ Unsupported
RevokeGrant	❌ Unsupported
ScheduleKeyDeletion	❌ Unsupported
Sign	✅ Supported
TagResource	✅ Supported
UntagResource	✅ Supported
UpdateAlias	✅ Supported
UpdateCustomKeyStore	❌ Unsupported
UntagResource	✅ Supported
UpdateKeyDescription	✅ Supported
UpdatePrimaryRegion	❌ Unsupported
Verify	✅ Supported
VerifyMac	✅ Supported

S3 Support

Most common operations of S3 are implemented. Remaining work:

• Versioning
• A bunch of metadata/usage APIs
• Policy/ACL is missing

S3 blocks are persisted, but metadata is not. This will be fixed in the future.

Click to expand the detailed support table

API	Support Status	Caveats/Notes
AbortMultipartUpload	✅ Supported
CompleteMultipartUpload	✅ Supported
CopyObject	✅ Supported
CreateBucket	✅ Supported
CreateMultipartUpload	✅ Supported
DeleteBucket	✅ Supported
DeleteBucketAnalyticsConfiguration	❌ Unsupported
DeleteBucketCors	❌ Unsupported
DeleteBucketEncryption	❌ Unsupported
DeleteBucketIntelligentTieringConfiguration	❌ Unsupported
DeleteBucketInventoryConfiguration	❌ Unsupported
DeleteBucketLifecycle	❌ Unsupported
DeleteBucketMetricsConfiguration	❌ Unsupported
DeleteBucketOwnershipControls	❌ Unsupported
DeleteBucketPolicy	❌ Unsupported
DeleteBucketReplication	❌ Unsupported
DeleteBucketTagging	✅ Supported
DeleteBucketWebsite	❌ Unsupported
DeleteObject	✅ Supported
DeleteObjects	✅ Supported
DeleteObjectTagging	✅ Supported
DeletePublicAccessBlock	❌ Unsupported
GetBucketAccelerateConfiguration	❌ Unsupported
GetBucketAcl	❌ Unsupported
GetBucketAnalyticsConfiguration	❌ Unsupported
GetBucketCors	❌ Unsupported
GetBucketEncryption	❌ Unsupported
GetBucketIntelligentTieringConfiguration	❌ Unsupported
GetBucketInventoryConfiguration	❌ Unsupported
GetBucketLifecycle	❌ Unsupported	Discouraged by AWS
GetBucketLifecycleConfiguration	❌ Unsupported
GetBucketLocation	❌ Unsupported
GetBucketLogging	❌ Unsupported
GetBucketMetricsConfiguration	❌ Unsupported
GetBucketNotification	❌ Unsupported	Discouraged by AWS. no longer used
GetBucketNotificationConfiguration	❌ Unsupported
GetBucketOwnershipControls	❌ Unsupported
GetBucketPolicy	❌ Unsupported
GetBucketPolicyStatus	❌ Unsupported
GetBucketReplication	❌ Unsupported
GetBucketRequestPayment	❌ Unsupported
GetBucketTagging	✅ Supported
GetBucketVersioning	❌ Unsupported
GetBucketWebsite	❌ Unsupported
GetObject	✅ Supported
GetObjectAcl	❌ Unsupported
GetObjectAttributes	❌ Unsupported
GetObjectLegalHold	❌ Unsupported
GetObjectLockConfiguration	❌ Unsupported
GetObjectRetention	❌ Unsupported
GetObjectTagging	✅ Supported
GetObjectTorrent	❌ Unsupported
GetPublicAccessBlock	❌ Unsupported
HeadBucket	✅ Supported
HeadObject	✅ Supported
ListBucketAnalyticsConfigurations	❌ Unsupported
ListBucketIntelligentTieringConfigurations	❌ Unsupported
ListBucketInventoryConfigurations	❌ Unsupported
ListBucketMetricsConfigurations	❌ Unsupported
ListBuckets	✅ Supported
ListMultipartUploads	❌ Unsupported	implement me!
ListObjects	❌ Unsupported	implement me!
ListObjectsV2	✅ Supported
ListObjectVersions	❌ Unsupported
ListParts	✅ Supported
PutBucketAccelerateConfiguration	❌ Unsupported
PutBucketAcl	❌ Unsupported
PutBucketAnalyticsConfiguration	❌ Unsupported
PutBucketCors	❌ Unsupported
PutBucketEncryption	❌ Unsupported
PutBucketIntelligentTieringConfiguration	❌ Unsupported
PutBucketInventoryConfiguration	❌ Unsupported
PutBucketLifecycle	❌ Unsupported	Deprecated
PutBucketLifecycleConfiguration	❌ Unsupported
PutBucketLogging	❌ Unsupported
PutBucketMetricsConfiguration	❌ Unsupported
PutBucketNotification	❌ Unsupported	No longer used
PutBucketNotificationConfiguration	❌ Unsupported
PutBucketOwnershipControls	❌ Unsupported
PutBucketPolicy	❌ Unsupported
PutBucketReplication	❌ Unsupported
PutBucketRequestPayment	❌ Unsupported
PutBucketTagging	✅ Supported
PutBucketVersioning	❌ Unsupported
PutBucketWebsite	❌ Unsupported
PutObject	✅ Supported
PutObjectAcl	❌ Unsupported
PutObjectLegalHold	❌ Unsupported
PutObjectLockConfiguration	❌ Unsupported
PutObjectRetention	❌ Unsupported
PutObjectTagging	✅ Supported
PutPublicAccessBlock	❌ Unsupported
RestoreObject	❌ Unsupported
SelectObjectContent	❌ Unsupported
UploadPart	✅ Supported
UploadPartCopy	❌ Unsupported
WriteGetObjectResponse	❌ Unsupported

SQS Support

SQS support is in in-progress.

Click to expand the detailed support table

API	Support Status	Caveats/Notes
AddPermission	❌ Unsupported
CancelMessageMoveTask	❌ Unsupported
ChangeMessageVisibility	✅ Supported
ChangeMessageVisibilityBatch	✅ Supported
CreateQueue	✅ Supported
DeleteMessage	✅ Supported
DeleteMessageBatch	✅ Supported
DeleteQueue	✅ Supported
GetQueueAttributes	✅ Supported
GetQueueUrl	✅ Supported
ListDeadLetterSourceQueues	❌ Unsupported
ListMessageMoveTasks	❌ Unsupported
ListQueues	✅ Supported
ListQueueTags	✅ Supported
PurgeQueue	❌ Unsupported
ReceiveMessage	✅ Supported	wait not supported
RemovePermission	❌ Unsupported
SendMessage	✅ Supported
SendMessageBatch	❌ Unsupported
SetQueueAttributes	✅ Supported	not all attributes
StartMessageMoveTask	❌ Unsupported
TagQueue	✅ Supported
UntagQueue	✅ Supported