AWS VolumeModificationSizeLimitExceeded

If you are dealing with big amounts of EBS volume and need to extend it you might face one day an AWS VolumeModificationSizeLimitExceeded error.

We are ocasionaly extending our EBS volumes, It is done by updating the CloudFormation for these instances and then there is a script (that was written by people much more AWS knowledgable then myself) that checks CloudFormation parameters and if it sees that volume size has increased it will do all the AWS and Linux “black magic” to actually extend the volumes and make the OS aware of it.

One happy day we were extending volumes for our 60 members strong fleet of EC2 instances from 2000GB to 3000GB.  So I’ve updated the CloudFormation and the extension kicked off.. On 51 out of 60 instances it has completed successfully, but on the rest I could see that cfn-script was complaining “An error occurred (VolumeModificationSizeLimitExceeded) when calling the ModifyVolume operation: You have exceeded your maximum gp2 volume modification storage limit of 100 TiB in this region. Please contact AWS Support to request an Elastic Block Store volume modification storage limit increase or retry once existing volume modifications have completed”

Apparently there is a limit on the “original” total volumes size that can be extended in one go and the default limit is 100TB and which we have hit:  52 x 2000GB = 101.56TB.

It is mention in https://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html as “Maximum modifying storage”

VolumeModificationSizeLimitExceeded
Maximum Modifying Storage

(As of December 2019) I couldn’t find where one can see the current limit and the only way to increase it is by logging a support case with AWS. By the way you will need to log a “generic” support case and not a “limit increase” case as this limit is not available for selection.

Also if you plan to extend from more then 300TB in one go you better contact AWS as soon as possible as here is what I’ve got from AWS  Support when trying to increase the value to 400TB

“With regards to your question, the 300TB is the value we can provide at this point because request with higher value requires further review/approval from our EBS team. The outcome depends on their review and we don’t guarantee whether the request will get approved or not.”

Ansible – loop over netsted variables

I’ve started using Ansible at my work, where we use it to deploy Splunk environments.

One of the things I needed to do is to provide a list of tcp ports to a “with_items” statement in a form of list.

I have this vars file and I needed to filter out only the TCP ports

So here is how you can do it (in 2 steps) in Ansible.

Of course you can replace the debug msg with whatever other action you need.
By the way if you know how to do it in single step please let me know.

Splunk – List REST API users and their IPs

Want to get a list REST API users and their IPs

Run this search

The limitation is if the users are going via a Load Balancer, you will see  Load Balancer’s IP as the clientip

Use Glide to create a catalog of books and movies from the Tim Ferris blog

So I was playing with web scraping a couple of years ago and scraped the list of Books, Movies and other items mentioned in Tim Ferris Blog and Podcast and yesterday I’ve somehow stumbled on the Glide. So I thought to myself, “why not try to use Glide to create a catalog of books and movies from the Tim Ferris blog?”

Continue reading Use Glide to create a catalog of books and movies from the Tim Ferris blog

Notes from SplunkLive! Sydney 2019

Notes from SplunkLive! Sydney 2019

I’ve had a chance to got SplunkLive! in Sydney this year.

It was freezing (by Sydney standards) 7.6 with winds which felt like -0.2 according to weatherzone app on my phone and my face.

So I wouldn’t have minded if the event turned out to be a total disaster, as long as they would have served coffee and it was warm inside, but it turned out to be quite interesting.

Continue reading Notes from SplunkLive! Sydney 2019

Python – Test Network Connection

The below will return True/False

Or you can use this version to return the Exception in case connection has failed

Python – Get local machine IP

Here is how to use Python – to Get local machine IP

 

Configure Splunk SSO with Auth0 as your identity provider

I had to work on Splunk SSO Integration and since had never touched SSO/SAML before, I wanted to play with it a little bit on my machine. I’ve decided to use Oath0 as my IdP

This tutorial is based on SAML SSO with Auth0 as Service Provider and as an Identity Provider, but the steps that are relevant to configuring an Auth0 tenant as the Service Provider (SP) are replaced with Splunk Configuration.

Continue reading Configure Splunk SSO with Auth0 as your identity provider

Return user roles in Auth0

I wanted to play with SAML Authentication in Splunk and decided to use Auth0 is my SAML Identity Provider (IdP).
Since i’ve never worked with Auth0 I just followed the SAML SSO with Auth0 as Service Provider and as an Identity Provider tutorial,, which worked well, but when I tried to use Splunk as Service Provider(SP), i.e. SAML service consumer, I noticed that roles are not returned by Auth0 SAML assertion, so I had to find a way to return user roles in Auth0 together with other user’s information.

Continue reading Return user roles in Auth0