Summertime, and the Livin’ is Data Catalog

A year ago I was at Credit Karma and was deep into development of their Data Explorer. I had executive sponsorship and all was well. The product launched live last August, so it was a huge success. There was a launch party, I was filmed in an interview and was also asked to write the companies November Engineering Blog. Cool beans.

A year later, and I’m in Oakland at LendUp, and I am happier than ever. I even had a short contract between Credit Karma and LendUp at Twitter – and I learned Collibra there – which was a gift. I’ve come to learn what is about to happen in the “data processing” world, and things are about to “Cross the Chasm”. For many companies it’s an unsettled time.

The theme of this story is that the world is about to experience a tsunami of new Data Privacy regulations the likes of that which has never been seen before. More than 50% of US companies are woefully unprepared. Some will wait too long and will panic by the fourth quarter. It will be in the news more and more each month.

Between GDPR and CCPA, any company that collects and processes personal information of pretty much any sort – including aggregated and derived data will be put under a microscope. It is not an easy or quick job to prepare for.

Blame it all on sloppy data management and dangerous personal data breaches – not just with social media companies, but also financial institutions.

Recently, Facebook, Google and other companies were fined heavily. Executives at Facebook have been issued subpoenas and have been called to testify.

The GOP and this White House are doing the same things the Bush administration did – reduce taxes for corporations and the wealthy, and deregulate like mad. All of this was the precursor to the worst recession since the great depression. I feel very strongly that we are only months away from the start of the next recession. But I digress.

One thing the GOP and this administration can’t do is deregulate or eliminate personal data privacy rights. This is because the GDPR is EU and CCPA is the State of California. What they can do is create a federal law so we don’t become overburdened with too many states enacting their version of CCPA.

I’ve worked with data my entire career, and as a DBA (Database Administrator) for the longest stretch. Managing data has been my life, man!

One thing I’ve learned is that GDPR and CCPA are the best things that can happen to any company that processes personal data. This is because the real dirty truth is that most companies don’t even know all the data that they have – where it is and what it is. This leads to exorbitant cloud storage costs. Trust me – I wrote one SQL script that ended up saving a company more than a million dollars in cloud costs! Some say it’s closer to 3 million, but the way these things are charged make it nearly impossible to even know what is costing what.

A year ago I cut my teeth designing, coding and launching the Credit Karma Data Explorer. It was a nice little Data Dictionary, but pales in comparison to what I’m designing and building now. When you add in the 3 intensive months working on Collibra at Twitter, I feel so far ahead and advanced than I was a year ago. I now know the big difference between a small Data Dictionary and a full blown Data Catalog – that can support the myriad data regulations – some that existed and some new.

This leads to me again having a spring in my step loving going to work with great people in a great Town and building the most important product of my career.

Best yet, this niche area that I am now entering the Expert Level just happens to be a whole new sub market in IT. I couldn’t be in a better place at the right time, and I’m proud to say I saw this coming more a year ago.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: