A little over two years ago I wrote about “My Six Month Journey as a Lead Web Developer”, this time I want to share with you how I ended up as a Director of Technology.
Note: I am not gloating over my promotion that went into effect June 2016, I just want to share my journey and also my professional failures along the way.
“If you ever find yourself writing a blog post on why PHP sucks, you are not ready.”
Sadly, Juozas is right! If you ever find yourself taking time out of your day to write why a specific programming language, web technology or framework “sucks”. You are not ready to take a bigger role than your current one; allow me to explain in depth.
Timeline: Late Spring 2015 to Late Summer 2015
My current employer hired me on April 25th, 2015 to do one job and that was to redesign their old website. If you’re curious here’s how it looked before; let’s go back in time through the Internet Archive Wayback Machine. Ha.
During this time period: early May 2015 through August 2015, the company began having IT infrastructure issues and I was asked to “help” resolve them. Let me make it clear that my employer did have an external IT partner, and this company had been contracted for over ten years. Here’s what I uncovered during the first week I began “helping” my co-workers with their workstation issues:
- The majority of the Dell workstations were running Windows 7 Pro, and were not up to date with current security patches.
- Most workstations were still running Java SE 6 update 75 (Last released 04/15/2014)
- Yes, all workstations had an all-in-one security end point, which I had never heard of before: VIPRE for Business.
- Most workstations at that time were about four years old, and in desperate need to be replaced.
- Our Dell PowerEdge R410 Server had so many hardware issues it was almost a joke.
- The server was running a VMware vSphere ESXi 5.1 virtualized Windows Server 2008 environment with little security updates.
- On May 27th, 2015 our server was hit with CryptoLocker, encrypting and locking all server files. Luckily, we did have an offsite backup and we were up and running in a few hours.
- Our SonicWALL TZ 105 security license had expired two months prior CryptoLocker hitting us, essentially the firewall was a sitting ethernet switch.
I believe you can see that our IT partner was all smoke and mirrors. I don’t want to blame management; just because they may read this ha! But let’s be honest, it’s pretty rare when management in a small business environment is technical.
Yes, my employer had to understand at least some level of technology needs and proper hardware maintenance. When I realized that our IT partner was a joke I had two options: One, depart from my position. Me thinking; “if this company doesn’t take serious their IT infrastructure they won’t take serious their web infrastructure”. Two, Boys Scout rule: Leave the camp ground cleaner than how you found it!
Well you guessed it, I took option two. And let me be honest it was not an easy decision to make, I had to sleep on it for a couple of days. Here’s why, I had committed myself to deliver a new redesigned website in a certain amount of time. Also, I knew once I opened “pandora’s box” I was bound to find more issues. Though, I just couldn’t let my employer keep throwing money away; it was unethical! So I began doing what us developers dread the most, document! I started by asking the front office girls to provide me with invoices sent by our IT partner. I then sat down at each workstation and took screenshots of the last updates made and security patches pending; sadly one workstation needed 145 security patches.
I then went into our “server room” and took pictures of the Dell PowerEdge R410 error messages, I also logged into our SonicWALL TZ 105 security admin console and took a screenshot when the security license had expired.
One weekend in late July 2015, I wrote a thorough internal report and I sent it directly to the President of the company. When I went in on the following work day, I was asked to break it down in a non-technical way. Once I began to speak in plain english; essentially telling upper management that they had been swindled for over ten years it was not easy for them. After our conversation I was asked to schedule a conference call with the owner of our IT partner, and once I began talking and showcasing all the issues. We received the following cold response:
There are simply too many cooks in the kitchen and we can’t guarantee the stability of your system when someone else has their hands in it.
Wow, after ten years of essentially a business partnership that was the response from our IT partner. When the call ended I had assumed new responsibilities, most would’ve been ecstatic about the opportunity to build something new from scratch; a blank canvas. However, I saw it with too many unknowns.
The first couple of days were not easy; by this time our former IT partner did not have the courtesy to provide us with logins and passwords. I began to create new accounts and negotiate with new security vendors to provide a solid security infrastructure. Also, we began to experience hardware issues with our Dell PowerEdge R410 Server. I contacted Dell to see if we could extend the hardware warranty.
Tip: Dell is pretty good in extending warranties to hardware in small business environments, keep that in mind!
However, once they ran both the service tag and serial numbers I was informed that the server was reported stolen in Japan. I can assure you that I am not joking! Apparently, this server had been pulled out from a cloud vendor datacenter in Japan and somehow ended up in a small manufacturing company in Southern California. I immediately notified upper management and they made contact with our former IT partner, which they provided eBay receipts; what a joke! Ultimately, the server failed and after several e-mails and memos to upper management I was granted a solid budget to upgrade the company’s network; sounds easy? Well here’s what I learned:
Don’t try to be a jack of all trades and master of none!
Timeline: Early Fall 2015 to Late Fall 2015
I have to be honest and say that I was way over my head with this new task (upgrading the entire company network). In early fall 2015, our office internet connection began to drop several times per day, we had a T1 connection that was painfully slow provided by TelePacific Communications. During this time I also began to fall behind in finishing the new website. My daily routine was to troubleshoot and support our entire office with network connectivity issues, this became a hassle and unnecessary stress. If you ever find yourself in a situation where you’re about to lose it, I wan’t you to know you’re not alone! Don’t be afraid to speak and ask for help; and that’s exactly what I did. I sat down with upper management in late September of 2015 and I requested outside help, they understood and my request was granted. I have to say by this time everyone in the office was complaining about either computer, internet or printing issues on a daily basis; I am sure this was the reason why my request was granted.
As a professional you need to understand your limits, and there is no shame in knowing where your knowledge has reached its limit!
Soon after, I began interviewing outside IT companies and one thing you will quickly find out, each company has a specific vendor list of both hardware and software they’ll try to pitch. Some prefer Dell, others HP and one Oracle/Solaris; I am not kidding you. My advice, pick a technology stack that can easily be picked up by another company or team in case it doesn’t workout; in our case I selected a Dell (workstations/servers) and Cisco (Managed Switched/Meraki Security Appliance) stack. Before you present what you believe to be a good fit to upper management make sure all questions are answered before they are even asked. This means finances. We picked one company (our current IT partner), and I am glad I was included in the decision making process. As the only technical individual in the company I am the go-to person when it comes to opening support tickets or making technology based decisions; at the end I am looking out for what is best for the company not our IT partner.
By late November 2015 most of our office workstations where upgraded; my boss and I are the only macheads in the office! And by mid December 2015, our internet connection was more reliable. I began to breathe a little smoother; ha. I quickly shifted all my time and focus to deliver what I had been hired to do in the first place, the website.
Before I move forward, this experience with both infrastructure and network hardware made me a better web developer. I had to quickly get out of my comfort zone; and I encourage everyone to do it time to time, it helps you grow professionally! To understand different aspects of technology, I was exposed to DNS, network latency, virtualization, security and both ISO and PCI compliance. This lead me to understand how the internet works a bit better.
Timeline: Early Spring 2016 Late Summer 2016
With my new knowledge, in early January 2016 I began to write code to be more efficient: “web friendly”, and what do I mean by this? I began to pay more attention to our Google Analytics in terms of mobile device visits, this includes both smartphones and tablets. About forty percent of our total web traffic comes from mobile devices, this led me to select a lightweight responsive design. Also, since our website mainly showcases media (images, PDF’s and CAD Drawings) I optimized each file to shrink their data size. In other words I came quite obsessed with technical web optimization. By mid February 2016, I was ready to launch the new redesign website. Days before launching, I ran into an issue; GoDaddy hosting.
Note: One thing about small business owners is they’re pretty loyal with companies or service providers they do business with. It is pretty hard to make a change in services or in my case web hosting; more on this later.
GoDaddy had been providing web hosting to the company since the early 2000’s; yeah loyalty! Prior to this web redesign, the company had done two previous web stack changes from ASP.NET (what where they thinking!) to LAMP. Getting back to the subject, the issue was ‘shared hosting’ there was no way that our current website would be able to thrive in a shared hosting environment, even with level 3 resources; all of you GoDaddy fans should know about this one, ha. Trying to explain to management about REST, HTTP and API requests as a possible factor to overloading our shared hosting server was not easy. My request to change to a dedicated server with a different hosting provider was not approved at the time.
Recipe for failure. (Downtime)
I understood upper management decision not to pull the trigger, I mean a few months earlier they had invested a lot of resources in the infrastructure upgrade, although I did not agree with the decision I had to accept it.
We launched the new redesigned website on Sunday, February 14th, 2016 before midnight. I knew by the following workday a long day awaited me, our site has stable high traffic, a lot of architects, home builders and contractors depend on our PDF files or CAD Drawings. Sure enough, by 09:00 AM uptime robot began sending me e-mails about downtime, and the phone calls began to come in by architects and contractors trying to access our PDF’s. I knew a call from upper management was coming, so I began to gather all the resources needed to explain the issue and provide a fix. This meant calling GoDaddy support and opening a ticket, requesting in writing a fix from their end (upgrading to a VPS or dedicated plan), and providing my solution switching to a new hosting provider. Once I had all the information needed I went into my ‘told-you-so’ mode and presented my case, after a brief meeting, I immediately switched to a new hosting provider and signed up for a CDN; more on CDN’s in a bit. We haven’t experienced any issues since.
Content Delivery Networks or CDN’s are a great way to free resources or bandwidth traffic from your server, some hosting providers offer VPS or dedicated server plans with a traffic limit usually between 2TB/5TB, very few offer unlimited, however, those who do their plans are pricy. Also, your assets are stored in geo-located data centers around the U.S. or world. This helps a lot with latency!
After an almost successful launch I was given another project, adding a shopping cart to our website. I will not go into great detail; cough, cough ‘confidentiality agreements’. Though, I can say I did spend a lot of time planning this project, my biggest concern at the time was how to implement security. I’m not talking about just slapping an SSL certificate but the overall security infrastructure, if something went wrong with data leaks I was the sole responsible person. This kept me up a few nights, and I took the most ethical approach.
Moving a bit forward. I received an e-mail from the President of the company pretty late on a June 2016 night, it was short only a few sentences but pretty powerful. I was offered to become the company’s first ever Director of Technology. I didn’t think twice and I accepted, opportunities like these don’t come often even at small companies.
By late July 2016, I began to oversee DevOps, E-Commerce and Marketing. My first task was to supervise another big project within the company, upgrading all workstations to Windows 10 Pro before the Microsoft July 29th, 2016 deadline. Our IT partner managed to complete this task with minimal downtime to operations by July 25th, 2016.
Timeline: Late Summer 2016 Late Fall 2016
By early September 2016 I began to receive pressure to complete our shopping cart integration; with my new title I requested interns. Ha. I hired three amazing interns to help me complete the shopping cart, this was a new experience for both the company and myself. Most companies hire interns to complete busy work; I personally experienced this early in my career. However, I brought them to gain real life experience in a fast paced environment, and it was great!
Around this time we began to experience internet connectivity issues, we quickly realized that as we grew our bandwidth needs had increased. I called TelePacific Communications to find out about our upgrade options and shortly after I submitted a RFP (Request For Proposal). Here’s where it gets a bit interesting, not long after submitting our RFP TelePacific Communications came back with adding more T1’s; nothing exciting here. However, they failed to provide other options more on this in a bit.
Our company has been a TelePacific Communications customer for over a decade; remember my loyalty note? I figured they would do their best to keep our business and provide decent pricing? Yeah, we found out the hard way that was not the case.
During the Holiday cheer of December 2016 a Cox Business representative walks in the office and leaves a lot of marketing material for me to read. I found out that our location was eligible for fiber and if we didn’t want to wait for the fiber lines to be pulled, our other option was fixed wireless. I quickly submitted an RFP and I received one in early January 2017.
A Bidding War
Timeline: Early Spring 2017 Mid Summer 2017
With the proposal from Cox Business in hand, I called our TelePacific Communications account manager with the news that we were thinking to take our business to Cox Business, immediately he requested to see the Cox Business proposal to match it. A bidding war began and at the end we stayed with TelePacific Communications. It came down to pricing!
At the end, it turned out to be one of the biggest cluster fucks! Sorry but I can’t find another way to express myself. Here’s where it went wrong, by late February 2017 our company President signed all documentation to move forward with a new VOIP Phone System, Fixed Wireless Internet and a 4G LTE failover; all equipment, training and installation included. What TelePacific Communications failed to disclose was a merger with another company, TelePacific Communications became TPX.
After the merger, TPX missed a lot of milestones in our installation and training; geez I don’t even want to remember! It was bad, and I felt guilty that our business operations were severely impacted; I was the one who pushed for this new service thus the guilt.
After the whole TPX issue; which I must say we are still waiting for them to make it right! Our shopping cart was launched in mid March 2017 with little fanfare. Once we began receiving online orders I believe upper management forgave me for the whole TPX issue. Ha.
A few months later, when the WannaCry ransomware became a global issue. Our security infrastructure passed its first true test since the CryptoLocker incident in May 2015. I can state that was a good day; small victories.
What am I up-to now? I can’t really provide any details but we are currently working on a mobile app for both smartphones and tablets (iOS/Android) with some augmented reality; I will write more about this in a later blog post. Stay tuned!
I want to finish by saying that my journey was not easy and it is not finished. I have learned a lot in the last two years and four months that I’ve been at the company, and I am thankful for the opportunities and patience that upper management has given me. As I mentioned earlier, I’m glad I was challenged both as an individual and professional; by stepping out my comfort zone. My advice to someone looking to pursue that professional title (Lead, Senior, Manager or Director) is not to lose sight of it.
It’s not the title that makes the person it’s the journey!
If you feel your current employer does not see your potential or your manager passes you for a promotion: Leave and stay positive, your time will come. I have grown professionally and have gained a lot of business insight at a small company, also, don’t be afraid to make a career change; trust me!