Splunk Connect for Kafka

My journey with Splunk Connect for Kafka.

Splunk Connect for Kafka (aka SC4K) allows to collect events from Kafka platform and send them to Splunk. While the sending part (to Splunk) was pretty straight forward to me, the collection part (from Kafka) was very new, as I’ve had no experience with Kafka eco-system. So I guess will start with it.

Continue reading Splunk Connect for Kafka

Splunk Certification Tracks

So here is my understanding of the current Splunk Certification Tracks.

Of course you can go to the “source”  https://www.splunk.com/en_us/training.html  , but may be that visual representation  will help someone

Splunk Certification Tracks
Splunk Certification Tracks

The dotted lines represent Recommended Prerequisites while solid lines are Mandatory Prerequisites for the different Splunk Certification Tracks

There is no meaning to the line colours by the way 🙂

AWS VolumeModificationSizeLimitExceeded

If you are dealing with big amounts of EBS volume and need to extend it you might face one day an AWS VolumeModificationSizeLimitExceeded error.

We are ocasionaly extending our EBS volumes, It is done by updating the CloudFormation for these instances and then there is a script (that was written by people much more AWS knowledgable then myself) that checks CloudFormation parameters and if it sees that volume size has increased it will do all the AWS and Linux “black magic” to actually extend the volumes and make the OS aware of it.

One happy day we were extending volumes for our 60 members strong fleet of EC2 instances from 2000GB to 3000GB.  So I’ve updated the CloudFormation and the extension kicked off.. On 51 out of 60 instances it has completed successfully, but on the rest I could see that cfn-script was complaining “An error occurred (VolumeModificationSizeLimitExceeded) when calling the ModifyVolume operation: You have exceeded your maximum gp2 volume modification storage limit of 100 TiB in this region. Please contact AWS Support to request an Elastic Block Store volume modification storage limit increase or retry once existing volume modifications have completed”

Apparently there is a limit on the “original” total volumes size that can be extended in one go and the default limit is 100TB and which we have hit:  52 x 2000GB = 101.56TB.

It is mention in https://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html as “Maximum modifying storage”

Maximum Modifying Storage

(As of December 2019) I couldn’t find where one can see the current limit and the only way to increase it is by logging a support case with AWS. By the way you will need to log a “generic” support case and not a “limit increase” case as this limit is not available for selection.

Also if you plan to extend from more then 300TB in one go you better contact AWS as soon as possible as here is what I’ve got from AWS  Support when trying to increase the value to 400TB

“With regards to your question, the 300TB is the value we can provide at this point because request with higher value requires further review/approval from our EBS team. The outcome depends on their review and we don’t guarantee whether the request will get approved or not.”

Ansible – loop over netsted variables

I’ve started using Ansible at my work, where we use it to deploy Splunk environments.

One of the things I needed to do is to provide a list of tcp ports to a “with_items” statement in a form of list.

I have this vars file and I needed to filter out only the TCP ports

So here is how you can do it (in 2 steps) in Ansible.

Of course you can replace the debug msg with whatever other action you need.
By the way if you know how to do it in single step please let me know.

Splunk – List REST API users and their IPs

Want to get a list REST API users and their IPs?

Run this search

The limitation is if the users are going via a Load Balancer, you will see  Load Balancer’s IP as the clientip

Use Glide to create a catalog of books and movies from the Tim Ferris blog

So I was playing with web scraping a couple of years ago and scraped the list of Books, Movies and other items mentioned in Tim Ferris Blog and Podcast and yesterday I’ve somehow stumbled on the Glide. So I thought to myself, “why not try to use Glide to create a catalog of books and movies from the Tim Ferris blog?”

Continue reading Use Glide to create a catalog of books and movies from the Tim Ferris blog

Notes from SplunkLive! Sydney 2019

Notes from SplunkLive! Sydney 2019

I’ve had a chance to got SplunkLive! in Sydney this year.

It was freezing (by Sydney standards) 7.6 with winds which felt like -0.2 according to weatherzone app on my phone and my face.

So I wouldn’t have minded if the event turned out to be a total disaster, as long as they would have served coffee and it was warm inside, but it turned out to be quite interesting.

Continue reading Notes from SplunkLive! Sydney 2019