CCPA: “GAME ON” in One Month

The California Consumer Protection Act is good for everyone, albeit painful at first. It is historic in the USA

On January 1, in just one month, a new sweeping data protection and privacy regulation goes into effect. If you have become increasingly annoyed that you can’t use your smart phone as a phone (I get non stop spam calls), or if you are constantly annoyed by increasingly large pop up advertisements that make the internet almost unusable, or most importantly, if you are tired of having your personal identification data exposed via breaches and hacks by large financial institutions – you now will have some “digital rights”.

The big losers will be “Marcomm” – Marketing Communication and Digital Marketers – who sell your personal information without you even knowing it, and social media companies like Facebook, Google, Twitter and even Amazon – all of which you don’t even know what they do with your data. If you are a resident of Califonia, you will now have 5 major “rights”:

  • Know what personal information is being collected about you
  • Access that information
  • Know if your personal information is disclosed, and with whom
  • Know if your personal information is sold and the right to opt out of the sale
  • Receive equal service and price whether or not you exercise their privacy rights

I can hear some whine that its just more “stupid regulation” and some derogatory BS about California. People like that will change their tune if their personal identity is hacked – like most of us have experienced.

There is a really wonderful side of this new regulation – companies that process data will FINALLY have to get their shit together. You would be amazed to know that most corporations – who moved off of mainframes to more distributed systems (which is pretty much every company in the world), and those who moved to the cloud, do not even know what data they have and where. These companies didn’t have to – there was zero legislation or regulations that forced what is now being called “Good Data Hygiene”.

There are some industries that are very much regulated – banks, loan companies, basically companies who move money around. There are state and federal laws. I don’t think online retail has been regulated nearly as much, and know Social Media is unregulated.

Way back in the late 1970’s – mainframes ruled the day – and IBM had a data catalog built in – which means you couldn’t load or share data without it being HEAVILY audited and tracked. As soon as we started going “distributed” – with systems interconnected and all over the place (in the 1980’s), we lost sight of where all of our data was – at least from a “single view” perspective.

I am sure someone will come up with a “CCPA Certified” icon for those who are certified and compliant

Our elections were hacked due to sleazy and sloppy data operations at many social media companies. I deep sixed all of my Social Media accounts after the 2016 Election – because I really do think our democracy has been compromised. So – when you combined that with several data breaches (Wells Fargo, Experian) – where my personal data was compromised – you start feeling vulnerable, and you start wanting these “data rights”.

Getting back to the “Good Data Hygiene” aspect – I have been in database development work for 40 years – I started out as Oracle’s 127th employee. As a DBA (Database Administrator), and now Data Architect, I have been appalled at how data is stashed on so many company data resources – and now mostly in the cloud – and how companies haven’t even known how much data they have and where it is. I won’t name names – but even trusted large corporations have this problem. Startups are steeped in this problem – the “data discipline” is horrible because IT management generally sucks. Its all about “Agile” – and “Continuous Code Delivery”. After about 10 years of Cloud Computing – I could scare the living shit out of you if you knew how data is slung around.

Starting about 2 years ago I designed a distributed Data Catalog:

https://engineering.creditkarma.com/credit-karma-data-explorer/

and then I had the pleasure of working with a great product called Collibra (at Twitter) – which has become a #1 Data Catalog solution – and which started in Belgium and “caught fire” after GDPR – the forerunner to CCPA, but for EU. In fact, I was the person who gave the final OK on Collibra at Twitter.

The problem at Credit Karma and Twitter was IT management. On one hand – I was able to influence big decisions and I did deliver. But then IT management basically dropped the ball and it became frustrating. It was typical IT management – “Hurry up and wait”. Its endemic in IT – and I have witnessed this every single place I have ever worked – there is a big rush, then a period where nothing is decided and everything “lolly gags” around. I swear – I’ve had to learn how to deal with getting my energy up to meet a fast and furious challenge – only to then sit around being bored as management doesn’t manage or lead. I know they will eventually get there – but I move fast and furious and take ownership – and do not do well when its “Hurry up and wait”.

Luckily, at LendUp, we have just enough people to get the job done – and because LendUp does Loans – we are already heavily regulated and so CCPA is just more of what we are used to. Also – because we are in one vertical market, we can move faster than say Credit Karma or Twitter – who both have massive amounts of data used for all kinds of different things. These companies have so many people that regardless of what they tell you about “flat orgnaization” – sorry – when you have thousands of employees spread all over the place, things slow down – and even get more political. With CCPA – this gets in the way for sure.

I have 2 patents in review right now – and both are based on the “last mile” of being able to manage data and have good “data hygiene”. They are both related to the “Holy Grail” – or area that my Credit Karma design or even Collibra is missing – and its a huge gaping hole – and that is File management. Everyone does database table management – but I could not find one Open Source or Commercial product that scans files – data and code – and generates meta data for our data catalog. I have FINALLY solved this problem – so now I can manage every data asset in any company. I have moved at lightning speed – and not only designed something much better than what I did at Credit Karma – but even have completely redesigned our Data Warehouse – which like every company I have worked at for the last 10 years – has needed to be done, and badly. In fact – what I have designed makes the data pipelines much simpler and for much less cost. It also has all of the CCPA data management features and data security and data quality baked in. Don’t get me started on Data Science, ML and AI – that whole area is the absolute WORST in Data Management. Data Scientists have been put on a God like pedestal, and I have NEVER seen such bad data hygiene. Here is how to trip up a Data Scientist – ask them if their data sets have a Data Quality Score – that they know what their Data Quality is and if their data quality gives them a “confidence factor” for their data models. They will turn white as a ghost. Data Science is used for “Personalization” – which means they design the algorithms that then push email to you to tell you what you should buy. Amazon is one of the better companies in this regard – they usually recommend something that is based on my likes. Other companies are trying to get as good. It takes tons of data – but that’s only because every data science organization I’ve seen is very sloppy. Good math – poor data hygiene.

This is a new area – but its something companies that process data should have been doing all along. The problem – IT Management did not see “Proper Data Hygiene” as a priority. It always pissed me off – since that has been in my blood since college. I really did believe in my textbooks!

With CCPA, the “Rush” is on – for people like me – its a Gold Rush – and one that I have been wanting to happen ever since we moved off mainframes and went distributed. In fact, I fully expect to ride this wave into my retirement – yeah – this is a new flood gate that opens January 1, and the main reason I wanted this idyllic week in an idyllic place is to get my batteries recharged and ready for what will be one hell of a New Years Onslaught.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: